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Abstract 


View-oriented group communication services are widely used for fault-tolerant distributed com- 
puting. For applications involving coherent data, it is important to know when a process has a 
primary view of the current group membership, usually defined as a view containing a majority 
out of a static universe of processes. For high availability in a system where processes can join and 
leave routinely, some researchers have suggested defining primary views dynamically, depending 
on having enough members in common with recent views. 

We present a new formal automaton specification, DvS, for the safety guarantees made by a 
practical group communication service providing a dynamic notion of primary view. The specifi- 
cation is a simple automaton, with only seven kinds of actions. We demonstrate the value of Dvs 
by showing both how it can be implemented and how it can be used in an application. Both pieces 
are shown formally, with assertional proofs. 

First, we present a distributed algorithm based on a group membership algorithm of Lotem, 
Keidar and Dolev; our version integrates communication with the membership service, uses infor- 
mation from the application processes saying when a view has been prepared for computation by 
the application, and uses a static view-oriented service internally. We prove that this algorithm 
implements DVS, in the sense of trace inclusion. 

Second, we present an application algorithm that is a variant of an algorithm of Amir, Dolev, 
Keidar, Melliar-Smith and Moser, modified to use DVS instead of a static service. We prove that. 
it implements a (non-group-oriented) totally-ordered-broadcast service. 


1 Introduction 


Applications designed for distributed systems must cope with failures, because in practical settings 
failures are very likely to happen in a distributed system. Coping with failures in a distributed sys- 
tem, however, is not an easy task. A convenient approach is that of using general purpose building 
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blocks that provide powerful distributed services and facilitate the construction of applications. 
One such building block is a view-oriented group communication service. 

Such a service enables application processes located at different nodes of a fault-prone dis- 
tributed network to operate collectively as a group, using the service to multicast messages to 
all members of the group. Examples of view-oriented group communication services are found in 
Isis [5], Transis [13], Totem [31], Newtop [16], Relacs [2], and Horus [34]. 

Solutions to practical, real world problems have benefited from group communication services. 
Isis-based software has been used to provide reliable group communication services for the New 
York Stock Exchange, for the Swiss Electronic Bourse and for the French Air Traffic Control 
System [6]. 

The heart of a group communication service is a group membership service, which provides 
each group member with a view of the group; a view includes a list of the processes that are 
members of the group. Views are crucial because they describe which processes participate in 
the computation and the system allow them to cooperate by guaranteeing that messages sent by 
a process in one view are delivered only to processes in the membership of that view, and only 
when they have the same view. Within each view, the service offers guarantees about the order 
and reliability of message delivery. Clearly each particular group communication service has its 
own set of properties that are offered to the user. A good survey of group communication services 
that provides a description of the guarantees made by each service is provided in [37]. 

For maximum usefulness, system building blocks should have simple and precise specifications 
of their guaranteed behavior. Producing good specifications for view-oriented group communi- 
cation services is difficult, because these services can be complicated, and because different such 
services provide different guarantees about safety, performance, and fault-tolerance. Examples 
of specifications for group membership services and view-oriented group communication services 
appear in [8, 4, 7, 9, 14, 17, 18, 19, 20, 30, 32, 35, 36]. 

In [17], we presented a specification, vs, for a view-oriented group communication service. 
This specification consists of a simple state machine expressing safety requirements, plus a timed 
trace property expressing conditional performance and fault-tolerance requirements. We used 
this specification as the basis for proving the correctness of a complex totally-ordered-broadcast 
algorithm based on [22, 1]. In ensuing work, Chockler has used a version of VS to model and verify 
an adaptive totally-ordered-broadcast algorithm [8], Lesley and Fekete [25] have proved that a 
version of an algorithm of Cristian and Schmuck [10] implements vs, and Khazan [23, 24] has 
used VS in the design of a load-balancing database algorithm. 

The VS service produces arbitrary views, with arbitrary membership sets. However, in many 
applications of vs, especially those with strong data coherence requirements, the application 
processes perform significant computations only when they have a special type of view called a 
primary view. For example, a replicated database application might only perform a read or write 
operation within a primary view, in order to ensure that each read receives the result of the last 
preceding write, in some consistent order of the operations. In this setting, a primary view is 
typically defined to be one whose membership comprises a majority of the universe of processes, 
or more generally, a quorum in a pre-defined quorum set in which all pairs of quorums intersect. 
The intersection property permits information flow from any previous primary to a newly formed 
one. 


Pre-defined quorum sets can yield efficient implementations in settings where the system 
configuration is relatively static. However, they work less well in settings where the configuration 
evolves over time, with processes joining and leaving the system. For such a setting, a dynamic 
notion of primary is needed, one that can change to conform with the system configuration. A 
dynamic notion of primary still needs to maintain some kind of intersection property, in order 
to permit enough information flow between successive primary views to achieve coherence. For 
example, each primary view might have to contain at least a majority of the processes in the 
previous primary view. Several dynamic voting schemes have been developed to define primaries 
adaptively [12, 15, 21, 26, 33]. 

In particular, Lotem, Keidar, and Dolev [26] have described an implementation of a group 
membership service that yields only primary views, according to a dynamic notion of primary. 
An interesting feature of their work is that it points out various subtleties of implementing such a 
membership service in a distributed manner — subtleties involving different opinions by different 
processes about what is the previous primary view. These difficulties have led to errors in some of 
the past work on dynamic voting. The algorithm of [26] copes with these subtleties by maintaining 
information about a collection of primary views that “might be” the previous primary view. The 
service deals with group membership only, and not with communication. Lotem et al. prove that 
their protocol satisfies the following condition on system executions: any two (primary) views 
that occur in an execution are linked by a chain of views where for every consecutive pair of views 
in the chain, there is some process that “knows” it belongs to both views. 


In this paper, we present a new formal automaton specification, DVS, for the safety guarantees 
made by a practical dynamic primary view group communication service. This service is inspired 
by the implementation of Lotem et al., but integrates communication with the group membership 
service. An important feature of our specification is our careful handling of the interface between 
the service and the application. When a new view starts, applications generally require some 
initial pre-processing, typically, an exchange of information, to prepare for ordinary computation. 
For example, processes in a coherent database application may need to exchange information 
about previous updates in order to bring everyone in the new view up to date. We expect each 
application process to indicate when it has completed this pre-processing for a new view v by 
“registering” the view. The DVS service uses registration information when it creates a new view 
v, in order to determine which previously-created views must satisfy the intersection property with 
respect to v. When all members have registered v, the application has gathered all information it 
needs from previous views, and the service no longer needs to ensure intersection in membership 
between views before v and any subsequent ones that are formed. 

Another feature of our specification, compared to that in [26], is that our specification is 
given as an automaton, which maintains state information about the views and the messages sent 
in each view. This global state can be used in invariants and abstraction functions, leading to 
assertional proofs of the correctness of implementations of Dvs, and also of applications built over 
Dvs. In contrast, Lotem et al. use a specification given in terms of the whole sequence of events 
in an execution, and therefore must use operational reasoning about complex sequences of events. 
Extensive experience with proofs of distributed algorithms suggests that assertional techniques 
are less error-prone; also they are more amenable to automated checking. 


We demonstrate the value of our DvS specification by showing both how it can be implemented 
and how it can be used in an application. Both pieces are shown formally, with assertional proofs. 

First, we consider an implementation that is a variant of the group membership algorithm of 
Lotem et al.; our variant integrates communication with the membership service, uses registration 
information from the application processes saying when a view has been prepared for computation 
by the application, and uses a static view-oriented service (a version of vs) internally. We prove 
that this algorithm implements Dvs, in the sense of trace inclusion. The proof uses a (single- 
valued) simulation relation and invariant assertions. The key to the proof is an invariant expressing 
a strong condition about nonempty intersections of views; the proof of this depends on relating a 
local check of majority intersection with known views to a global check of nonempty intersection 
with existing views. 

Second, we consider an application algorithm that is a variant of an algorithm in [22, 1, 17], 
modified to use DVS instead of a static view-oriented service. The modified algorithm uses the 
registration capability to tell the Dvs service that information has been successfully exchanged at 
the beginning of a new view. We show that it implements a (non-group-oriented) totally-ordered- 
broadcast service. This proof also uses a simulation relation and invariant assertions. 

We have designed our DVS specification to express the guarantees that we think are useful in 
verifying correctness of applications that use the service. 


Among previous work, two different sorts of specifications for a primary group service are 
notable. Work by Ricciardi and others [36] is expressed in terms of temporal logic on consistent 
cuts; the idea of their specification is that on any cut, there are no disjoint sets of processes 
such that each set is collectively aware of no members outside that set. Lotem et al. [26] use a 
property of an execution, which was previously defined by Cristian [9] for majority groups: any 
two (primary) views are linked by a chain of views where every consecutive pair of views includes 
a process that “knows” it belongs to both views. As far as we know, these previous specifications 
have not been used to verify properties of applications running above them. 

Our specification omits some properties of existing dynamic primary view management algo- 
rithms. For example, Isis [5] guarantees that processes that move together from one view to the 
next receive exactly the same messages in the first view. Guaranteeing this property requires state 
exchange within the view management service. This property is not needed to verify properties 
of applications such as the one giving a totally-ordered broadcast. Also, our service provides no 
explicit support for application-level state exchange. Systems like Isis do provide such support, 
by allowing application-level state exchange messages to be piggybacked on the lower-level state 
exchange messages. 


In Section 2 we present our mathematical notation. The DVS service is presented in Section 3, 
and its implementation in Section 4. In Section 5 we use DVS to implement a totally ordered 
broadcast service. Section 6 contains some conclusions. 


2 Mathematical foundations 


2.1 Sets, functions, sequences 


We write » for the empty sequence. If a is a sequence then |a| denotes the length of a. If a is 
a sequence and 1 <i < j < |a| then a(i) denotes the ith element of a and a(i..j) denotes the 
subsequence a(z),a(i + 1),...,a(7) of a. The head of a nonempty sequence a is a(1). A sequence 
can be used as a queue: the append operation modifies the sequence by concatenating it with a 
new element and the remove operation modifies the sequence by deleting its head. 

If a and b are sequences, a finite, then a+b denotes the concatenation of a and b. We sometimes 
abuse this notation by letting a or b be a single element. We say that sequence a is a prefix of 
sequence 6, written a < b, provided that there exists c such that atc = b. A collection A of 
sequences is consistent provided that a < 6 or b < a for all a,b € A. If A is a consistent collection 
of sequences, we define /ub(A) to be the minimum sequence 6b such that a < 6 for alla € A. 

If S is aset, then segof (5) denotes the set of all finite sequences of elements of S. Ifa € seqof(S) 
and f is a partial function from S to T whose domain includes the set of all elements of S 
appearing in a, then applytoall(f,a) denotes the sequence 6 such that length(b) = length(a) and, 
for 1 < length(b), b(t) = f (a(t). 

If S is a set, the notation S, refers to the set SU{L}. Whenever S is ordered, we order S_ 
by extending the order on S$, and making less than all elements of S. If R is a binary relation, 
then we define dom(R), the domain of R, to be the set (without repetitions), of first elements of 
the ordered pairs comprising relation R. If f is a partial function from S to T, and (s,t) € Sx T, 
then f @ (s,t) is defined to be the partial function that is identical to f except that f(s) = t. 

P denotes the universe of all processors,! and M the universe of all possible messages. G is 
a totally ordered set of identifiers used to distinguish views, with a distinguished least element 
go. A view v = (g,P) consists of a view identifier g € G and a nonempty membership set 
P CP; we write v.id and v.set to denote the view identifier and membership set components of 
v, respectively. V denotes the set of all views, and vp = (go, Po) is a distinguished initial view. 


2.2 I/O automata 


We describe our services and algorithms using the I/O automaton model of Lynch and Tuttle [28] 
(without fairness). The model and its proof methods are described in Chapter 8 of [27]. 

An execution fragment of an I/O automaton is an alternating sequence of states and actions 
consistent with the transition relation. An execution is an execution fragment that begins with 
a start state. The trace of an execution fragment a is the subsequence of a consisting of all the 
external actions. The external behavior of an I/O automaton is captured by the set of traces 
generated by its executions. 

Execution fragments can be concatenated. Definitions of composition for I/O automata appear 
in Chapter 8 of [27], along with theorems showing that composition respects the external behavior. 
Invariant and simulation methods for these models are also presented in that chapter. 


‘We use “processor” and “process” interchangeably, since the difference is immaterial in the context of this 
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paper. 


3 The DVS specification 


We now present DVS, our specification for a dynamic primary view group communication service. 

The DVS service works as follows. Each client of the service has a “current” view of the group 
of processes. A process can send a message to all other members of its current view and the 
service guarantees that messages sent within a view are delivered only within that view and each 
member of the view receives messages in the same order as other members. However, not all 
messages need to be delivered to all members. The service also provides a “safe” notification 
for a particular message m that tells the recipient that message m has been received by all the 
members of the current view. New views are announced to all members of the new view and 
they are guaranteed to be “primary” views. Primary views are defined according to a dynamic 
notion [21]: a new primary needs to contain a majority of the members of the previous primary. 
The DVS service allows the clients to “register” a new view after completing the pre-processing 
for that view. 

The specification is given in Figure 1. In this specification, M. C M denotes the set of mes- 
sages that clients may use for communication. The most interesting part of the Dvs specification 
is the transition definition for pvs-cREATEVIEW(v). The precondition specifies the properties that a 
view must satisfy in order to be considered primary. For example, the precondition says that v.set 
must intersect the membership set of all previously-created smaller-id views w for which there is 
no intervening totally registered view — that is, the set of all “possible previous primary views”. 
Since (for convenience) we allow out-of-order view creation in DVS, we also include a symmetric 
condition for previously-created larger-id views. All created views are recorded in created. 

DVS informs its clients of view changes using Dvs-NEWvIEW((g, P))» actions; such an action in- 
forms processor p that the view identifier g is associated with membership set P and that the 
current group of processors connected to p is P. After any finite execution, we define the current 
view at p to be the argument v in the last pvs-NEwviEw(v), event, if any, otherwise it is the initial 
view vo for processors in Py and is undefined for other processors. Even though views can be 
created out of view id order, the notification to each client is consistent with that order. Not 
every client needs to see every view. The variable attempted records, for each view, which process 
have been notified of that view. Variable attempted is only used in the proof. 

With the pvs-REGISTER, action, the client at p informs the service that it has obtained whatever 
information the application needs to begin operating in the new view v. For many applications, 
this will mean that p has received messages from every other member of view v, reporting its state 
at the start of v. The variable registered records, for each view, which process have registered 
that view. Variable registered is only used in the proof. 

DVS allows a processor p to broadcast a message m using a Dvs-GPsND(m), action, and delivers 
the message to a processor q using a Dvs-GPRCV(m),,q action. DVS also uses a DvS-SAFE(m)p,q action 
to report to processor q that the earlier message m from p has been delivered to all members of 
the current view of g. DVS guarantees that messages sent by a processor p when the current view 
of p is v are delivered only within view v (i.e., only to processors in v.set whose current view is 
v). Moreover, each processor receives messages in the same order as other processor and without 
gaps in the sequence of received messages; however, a processor may receive only a prefix of the 
sequence of messages received by another processor. Variables queue, pending, next and next-safe 
are used for handling the messages. Their use should be clear from the code. 


Signature: 


Input: DVS-GPSND(m)p, m€ Me, pE P 
DVS-REGISTERy, p € P 

Internal: DVS-CREATEVIEW(v), v€ V 
DVS-ORDER(m, p,g), ME Me, DEP, GEG 


Output: | DVS-GPRCV(m)p,q, m € Me, p,q € P 
DVS-SAFE(M)p,qg, M € Me, p,q € P 
DVS-NEWVIEW(v)p, vu € V, p € v.set 


State: 
created € 2”, init {vo} 
for each p € P: 
current-viewtd|p] € G., init go if p € Po, else 
for each g € G: 
queue[g] € segof(M-. x P), init » 
attempted[g] € 2”, init Py if g = go, {} else 
registered|g] € 2”, init Po if g = go, {} else 


for each p€ P, g € G: 
pending|p, g] € seqof(M.-), init » 
next[p, g] € N°, init 1 
next-safe[p, g] € N°, init 1 


Transitions: 
internal DVS-CREATEVIEW(v) internal DvS-ORDER(m, p, g) 
Pre: Vw € created : v.id # w.id Pre: m is head of pendinglp, g] 
Vw € created : Eff: remove head of pending[p, g] 
dar € TotReg : w.id < xvid < v.id append (m,p) to queuelg] 
or dx € TotReg : v.id < x.id < w.id 
or v.set Nw.set # {} output DVS-GPRCV(m)p,q, Choose g 
Eff: created := created U {v} Pre: g = current-viewid|q] 
quenclg](nextla, ol) = (r,p) 
output DVS-NEWVIEW(v)» Eff: nezilg, g] :-= next[g, g] +1 
Pre: v € created 
v.id > current-viewtd[p| output DVS-SAFE(m) p,q, choose g, P 
Eff: current-viewid[p] := v.id Pre: g = current-viewid|q] 
attempted|v.id] := attempted|v.id] U {p} (g, P) € created 
quenelg](next-safela, gl) = (m,>) 
input DVS-REGISTER, for all r € P: 
Eff: if current-viewid[p] A L then nezt[r, g] > next-safelq, g] 
registered[current-viewid[p]] := Eff: nezt-safelq, g] := next-safelg, g] +1 


registered|current-viewid|p]| U {p} 
input DVS-GPSND(m)p 


Eff: if current-viewid[p] # L then 
append m to pending[p, current-viewid[p]| 


Figure 1: The DVS service 


We define the following derived variables: 


Att € 2”, defined as {v € created | attempted|v.id] 4 {}} 
TotAtt € 2”, defined as {v € created | v.set C attempted|v.id]} 
Reg € 2”, defined as {v € created | registered|v.id] # {}} 
TotReg € 2”, defined as {v € created | v.set C registered|v.id]} 


Informally, a view belongs to the set Att if it has been reported to at least one member of the 
view (we say that it is attempted). A view belongs to the set JotAtt if it has been reported to all 
members of the view (we say that the view is totally attempted). Similarly, a view belongs to the 
set Reg if at least one member of the view has registered the view (we say that it is registered) 
and belongs to the set JotReg, if all members of the view have registered the view (we say that 
the view is totally registered). 


We close this section with some invariants giving properties of Dvs. 

The first one is a trivial invariant which follows directly from the definition of the sets 
Att, Tot Att, Reg and TotReg. 

Invariant 3.1 (Dvs) 
In any reachable state, TotAtt C Att, TotReg C Reg, Reg C Att, and TotReg © Tot Att. 

The next invariant is a basic invariant saying that if a process p has attempted a view v whose 
identifier is g then the current view of p is either v itself or a view with an identifier greater than 
g: 

Invariant 3.2 (Dvs) 
In any reachable state if p € attempted|g] then current-viewid[p| > g. 


Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. In the initial state p € attempted|g| implies that p € Pp and 
g = go. For p € Py we have that current-viewid|p] = go and hence the invariant is true. 

For the inductive step assume the invariant is true in s. We need to prove that it is true 
in s’ for any possible step (s,7,s’). The only step that changes attempted and current-viewid is 
1] =DVS-CREATEVIEW(v)p. By the precondition of 7 we have that for any g for which p € attempted|g] 
it holds v.id > current-viewid[p] and by the code we have that the new value of current-viewid[p] 
is v.td. Hence the invariant is still true. O 


Invariant 3.3 expresses the key intersection property guaranteed by Dvs; this is weaker than 
the intersection property required by static definitions of primary views, which says that all 
primary components must intersect. This invariant is our version of the correctness requirement 
for dynamic view services that two consecutive primary views intersect. 


Invariant 3.3 (Dvs) 
In any reachable state, if v,w € created, v.id < w.id, and there is no x € TotReg such that 
v.id < x.id < w.id, then v.set ON w.set # {}. 


Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. In the initial state created = {vo} and thus the invariant is 
vacuously true. 


For the inductive step assume the invariant is true in s. We need to prove that it is true in s’ 
for any possible step (s,7,s’). The only steps that can change the hypothesis from false to true 
are DVS-CREATEVIEW(v) and Dvs-CREATEVIEW(w). The preconditions of these actions show that the 
needed conclusion holds. No step changes the conclusion from true to false. O 


Invariant 3.4 says that if a view w is totally attempted, then any earlier view v has a member 
whose current view is later than v. 


Invariant 3.4 (Dvs) 
In any reachable state, if v € created, w € TotAtt, and v.id < w.id, then there exists p € v.set 
with current-viewid|p] > v.id. 


Proof: Consider any particular reachable state. Assume that v € created, w € TJotAtt, and 
v.id < w.id. Then let y be the view in JotAtt having the smallest viewid strictly greater than 
v.id. Then there is no x € JotAtt with v.id < x.id < y.id. Then Invariant 3.1 implies that there 
is no x € TotReg with v.id < x.id < y.id. Then Invariant 3.3 implies that v.set Ny.set # {}. Let 
p € v.setNy.set, then p € attempted[y.id]. Then Invariant 3.2 implies that current-viewid[p| > y.id. 
This implies current-viewid|p] > v.id. O 


4 An implementation of Dvs 


We now present an algorithm that implements the DVS service specification and reason about 
its correctness. Our implementation uses as a building block the group communication service 
vs [17], and it uses the ideas from [26]. The overall system is comprised of the automata vs- 
TO-DVS,, for each p € P, and the VS service. We call this system DVS-IMPL and we illustrate 
it in Figure 2. Formally, the DVS-IMPL system is the composition of all vS-TO-Dvs, automata 
(presented in Section 4.2) and the vs automaton (given in Section 4.1). We show that Dvs-IMPL 
is a formal implementation of the DvS service in the sense of the trace inclusion, that is we prove 
that any trace of the Dvs implementation is a trace of the DvS specification (Section 4.3). 


DVS-IMPL 


Figure 2: The DVS-IMPL system. 


4.1 The VS specification 


The vs service [17] is a group communication service that is similar to Dvs except that vs does 
not provide support for primary views. The DVS service thus can be seen as an augmented version 


of the vs service designed to provide support for primary views. Due to the similarity of the two 
services, VS is a convenient building block for Dvs. The specification for the VS service is given in 
Figure 3. To avoid a complete restatement we refer the reader to [17] for an informal description 
of the service. 


Signature: 
Input: VS-GPSND(m)p, mE M, pE P Output: VS-GPRCV(m)p,¢, m € M, p,qeEP 
Internal: VS-CREATEVIEW(v), v € V VS-SAFE(™M)p,q, m € M, p,q € P, 
VS-ORDER(m, p,g), mE M, pEP, Gg EG VS-NEWVIEW(v)p, v € V, p € v.set 
State: 
created € 2”, init {vo} for each pE P,g EG: 
for each p € P: pending|p, g] € seqgof (M), init » 
current-viewtd|p] € G., init go if p € Po, else next[p, g] € N°, init 1 
for each g € G: next-safe[p, g] € N°, init 1 
queue[g] € segof(M x P), init » 
Transitions: 
internal vs-CREATEVIEW(v) output VS-GPRCV(m)p,q, Choose g 
Pre: Vw € created: v.id > w.id Pre: g#L 
Eff: created := created U{v} g = current-viewid|q] 
quenelg] (nest{g, gl) = (m,p) 
output VS-NEWVIEW(v)p Eff: nesi{q, g] := nectlg, g] +1 
Pre: v € created 
v.id > current-viewtd[p] output VS-SAFE(m)p,q, choose g, P 
Eff: current-viewtd|p] := v.id Pre: g # L 
g = current-viewid[q] 
input VS-GPSND(m)p (g, P) € created 
Eff: if current-viewid[p] # L then queue[g|(nezt-safelg, g]) = (m, p) 
append m to pending|[p, current-viewid[p]| for all r € P: 
nezt[r, g] > next-safelq, g] 
internal vs-ORDER(m, p, 9) Eff: nezt-safelq, g] := next-safelq, g] +1 


Pre: m is head of pending[p, g] 
Eff: remove head of pending[p, g] 
append (m,p) to queuelg] 


Figure 3: The VS service 


As also reasoned in [17], the fact that vs allows views to be created only in the order of view 
identifier is not significant: weakening this requirement to allow out-of-order view creation does 
not change the external behavior, because vs-NEwvisw actions are constrained to occur in such a 
way that views are always delivered in the order of view identifiers. 


We rely on the following safety properties of the vs service [17]: 
e New views are reported in increasing order of view identifier (monotone views property); 
e Messages sent in a view are delivered only within that view (view synchrony property); 
e The sequences of messages delivered in a view at any two processors are such that one sequence 


is a prefix of the other (prefix order property). 
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The following invariant holds. 


Invariant 4.1 (vs) 
In any reachable state, if v,v' € created and v.id = v' id, then v =". 


4.2. The Dvs implementation algorithm and the Dvs-IMPL system 


The Dvs implementation algorithm is given in terms of the automaton VS-TO-DVS»p, where p € P, 
in Figure 4. VS-TO-DVS, acts as a “filter”, receiving vs-Newview inputs from the underlying 
VS service and deciding whether to accept the proposed views as primary views. If vS-TO-DvS, 
decides to accept some such view v, it “attempts” the view by performing a pvs-NEWviEWw(v) output. 
For each v, we think of the DVS internal pvs-crEaTEVIEW(v) action as occurring at the time of the 
first pvs-NEWVIEW(v) event. 

VS-TO-DVS, uses special messages, tagged either with “info” or “registered”. Thus, we use 
M=M_.U ({ “info”} x V x 2”) U { “registered”}, where M, is the set of all client messages and 
M is the universe of all messages. The state variables attempted, reg, and info-sent are auxiliary 
— they are not needed for the algorithm, and are only used in the proofs. 

According to the Dvs specification, the algorithm is supposed to guarantee nonempty inter- 
section of each newly created primary view v with any previously created view w having no 
intervening totally registered view — this is a global condition involving nonempty intersection 
of view sets. The vS-TO-DVS, processors, however, do not have accurate knowledge of which 
primary views have been created by other processors, nor of which views are totally registered. 
Therefore, the processors employ a local check of majority intersection with known views, rather 
than a global check of nonempty intersection with existing views. Specifically, each VS-TO-DVSp 
keeps track of an “active” view act, which is the latest view that it knows to be totally registered, 
plus a set of “ambiguous” views amb, which are all the views that it knows have been attempted 
(i.e., have had a pvs-Newview action performed someplace), and whose identifiers are greater than 
act.id. We define use = {act} U amb. When VS-TO-DVS, receives a vs-NEWvIEW(v) input, it sends 
out “info” messages containing its current act and amb values to all the other processors in the 
new view, using the VS service, and then waits to receive corresponding “info” messages for view 
v from all the other processors in the view. After receiving this information (and updating its 
own act and amb accordingly), vS-TO-Dvs, checks that v has a majority intersection with each 
view in use. If so, VS-TO-Dvs, performs a Dvs-NEWvIEW, Output. 

Following the pvs-NewviEw, even, the clients of the communication system can exchange state 
information as needed for processing in view v. When the client at p has obtained enough 
information, it “registers” the view by means of action Dvs-REGISTER,, which causes processor p to 
send “registered” messages to the other members. When a processor receives “registered” messages 
for a view v from all members, it may perform garbage collection by discarding information about 
views with identifiers smaller than that of v. VS-TO-DVS uses VS to send and receive messages. 

The system DVS-IMPL is defined as the composition of all the vS-TO-Dvs, automata and the 
VS service, with all the external actions of vS hidden. 

We define four derived variables for DvS-IMPL analogous to those of DVS, indicating the 
attempted, totally attempted, registered, and totally registered views, respectively. They are: 


e Att = {v € created | (Ap € v.set)v € attempted,}; 
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Signature: 


Input: DVS-GPSND(m)p, m € M- 
DVS-REGISTERp 
VS-NEWVIEW(v)p, v € V, p € v.set 
VS-GPROV(m)q,p, MEM, qEP 
VS-SAFE(m)q,p, mE M, gE P 
State: 


cur € V,, init vo if p € Po, 1 else 
client-cur € V_, init vo if p € Po, 1 else 
act € Y, init vo 
amb € 2”, init {} 
attempted € 2”, init {vo} if p € Po, {} else 
for each g EG 
msgs-to-vs[g] € seqof(M), init » 
msgs-from-us|g] € seqof(Me. x P), init » 
safe-from-vs|g] € seqof(Me- x P), init » 


Internal 
Output: 


reg[g| a bool, init true if p € Po and g = go, false else 


info-sent[g] € (V x 2”) 1, init L 


Transitions: 


input VS-NEWVIEW(v)p 
Eff} cur:=v 

append ( “info”, act, amb) to 
msgs-to-vs|cur.id] 

info-sent|cur.id| := (act, amb) 


input vs-GPRCV(( “info”, v, V))q,p 
Eff: info-rcud|q, cur.id] := (v,V) 
if v.id > act.id then act:=v 
amb := {w € ambU V | w.id > act.id} 


input vs-SAFE({ “info”, v, V))a,p 
Eff: none 


output DVS-NEWVIEW(v)p 
Pre: v = cur 
v.id > client-cur.id 
Vq € v.set,g#p: info-rcvdiq,v.id]  L 
Vw € use: |v.setN w.set| > |w.set|/2 
amb := ambU {v} 
attempted := attempted U {v} 
client-cur := v 


Eff: 


input DVS-REGISTERp 
Eff: if client-cur #£ L then 
reg|client-cur] := true 
append ( “registered”) to 
msgs-to-vs| client-cur.id] 


input VS-GPRCV(( “registered”))¢,p 
Eff: rcvd-rgst|cur.id, q] := true 


: DVS-GARBAGE-COLLECT(v)p, uv € V 
VS-GPSND(m)p, m € M 
DVS-NEWVIEW(v)p, v € V, p € v.set 
DVS-GPRCV(m)q,p, mE Me, EP 
DVS-SAFE(m)q,p, ME Me, gE P 


for each g EG, qEP 
info-revd|q, g| € (V x 2”)., init L 
rcvd-rgst|q, g] a bool, init false 


Derived variables 
use € 2”, defined as use = {act} U amb 


input VS-SAFE(( “registered”) )q,p 
Eff: none 


internal DvVS-GARBAGE-COLLECT(v) p 
Pre: Vq € v.set : rcud-rgst{q, v.id] = true 
v.id > act.id 
Eff: act:=v 
amb := {w € amb | w.id > act.id} 


input DVS-GPSND(m)p 
Eff: if client-cur.idp # L then 
append m to msqgs-to-vs|client-cur.id] 


output VS-GPSND(m)p 
Pre: m is head of msgs-to-vs[cur.id] 
Eff: remove head of msqs-to-vs[cur.id] 


input VS-GPRCV(m)q,p, where m € M, 
Eff: append (m,q) to msqgs-from-vs|[cur.id] 


output DVS-GPRCV(m)q,p 
Pre: (m,q) is head of msgs-from-vs[client-cur.id] 
Eff: remove head of msgs-from-vs[client-cur.id] 


input VS-SAFE(m)¢,p, where m € M, 
Eff: append (m,q) to safe-from-vs[cur.id] 


output DVS-SAFE(m)p 
Pre: (m,q) is head of safe-from-vs|client-cur.id] 
Eff: remove head of safe-from-vs[client-cur.id] 


Figure 4: VS-TO-DVSp 
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e TotAtt = {v € created | (Vp € v.set)v € attempted, }; 
e Reg = {v € created | (Ap € v.set)reg|v.id], = true}; and 


e JotReg = {uv € created | (Vp € v.set)reg|v.id]p = true}. 
Another derived variable, use, is defined in the code of VS-TO-DVSp. 


4.3. Correctness of the DVS-IMPL system 


We prove that DVS-IMPL implements DvS using a forward simulation argument [29] by providing 
an abstraction function that maps states of DVS-IMPL to states of DvS and that leads to the main 
observation that each trace of DVS-IMPL is a trace of DvS. We present such an abstraction function 
in Section 4.3.2. 

Section 4.3.1 unveils a series of invariants of DVS-IMPL culminating in Invariant 4.17 and 
Invariant 4.18. The local condition requiring a majority intersection is captured by Invariant 4.17. 
Invariant 4.18 states that any two attempted views that have no intervening totally registered 
view have at least one member in common. This is the global condition on nonempty intersection 
that we have discussed in the previous section. These invariants are then used in the proof that 
DVS-IMPL implements DVS in Section 4.3.2. 


4.3.1 Invariants 


We begin with invariants that state simple facts about Dvs and then proceed to more complex 
ones ending with the key invariant about the global condition on nonempty intersections. 


Invariant 4.2 (DVvS-IMPL) 
In any reachable state, if curp # L then current-viewid[p] = cur.idp. 


Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. Fix p. In the initial state we have that curp = L. 
For the inductive step assume the invariant is true in s. We need to prove that it is true in s’ 
for any possible step (s,7, 8’). Fix p. We prove the invariant considering each possible action 7. 
1. 7 = VS-NEWVIEW(v)p. 
By the code of a in vs, we have that current-viewid[p] = v.id. By the code of 7 in DVS-IMPL, 
we have that cur.id, = v.id. 
2. Other actions. 


Variables current-viewid|p| and cur.idy are not modified. Hence the assertion cannot be made 
false. 


Invariant 4.3 (DVS-IMPL) 
In any reachable state, if v € attempted, then client-cur.idy = v.id. 
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Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. Fix v,p. In the initial state we have that attempted, = {vo} for 
p © Po and attempted, = 1 for p ¢ Po. So assume that v = vp and p € Py. Then client-curp = vo. 
Hence the invariant is true. 

For the inductive step assume the invariant is true in s. We need to prove that it is true in 
s’ for any possible step (s,7,s’). Fix v,p and assume that v € s’.attempted,. We distinguish two 
possible cases. 


1. v € s.attempted,. 
By the inductive hypothesis we have that s.client-cur, > v.id. By the monotonicity of 
client-cury we have that s'.client-curp > s.client-curp. 


2. v ¢ s.attempted,. 


Then it must be 7 =pvs-NEWvIEW(v),. The invariant follows from the code which sets client-curp 
to v. 


Invariant 4.4 (DVS-IMPL) 
In any reachable state, if v € info-sent|g]p = (x,X) then cur.idy > g. 


Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. Fix v,p. In the initial state we have that info-sent, = 1 and 
thus the invariant is vacuously true. 
For the inductive step assume the invariant is true in s. We need to prove that it is true in 
s' for any possible step (s, 7,8’). Fix p,g,x,X and assume that s’.info-sent[g], = («,X). We 
distinguish two possible cases. 
1. s.info-sent[g|, = (x, X) 
By the inductive hypothesis we have that s.cury > g. By the monotonicity of cur, we have 
that 8! .CUTp > 8.cury. Hence the invariant is true. 
2. s.info-sent|g|, A (x, X) 


Then it must be 7 =vs-NewviEw(v), and g = v.id = s'.act.idp. Action vs-NEWVIEW(v)» sets s’.cur 
to v, so s’.cur.id = g. 


Invariant 4.5 (DVS-IMPL) 
In any reachable state: 


1. vo € TotReg. 


2. go < vid for all v € created. 
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Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. Part 1 is true because in then initial state every processor 
p © Po has reg[go] = true. Part 2 is true because the only view in created is vo. 

For the inductive step assume the invariant is true in s. We need to prove that it is true in s’ 
for any possible step (s, 7, s’). 

Consider Part 1 first. No view is ever removed from JotReg. Hence no step can make the 
assertion false. Consider Part 2 now. Fix v and assume that v € s’.created. We distinguish two 
cases. 


1. v € s.created. 


Then the assertion follows from the inductive hypothesis. 


2. v € s.created. 


It must be 7=vs-crEATEVIEW(v)». By the precondition of this action we have that v.id > w.id 
for all w © s.created. By the inductive hypothesis gy < w.id for all w © s.created. Since 
s'.created = s.created U {v}, it follows that go < w.id for all w € s’.created. 


Invariant 4.6 (DVS-IMPL) 
In any reachable state, if rcvd-rgst[q, v.id]p A L then curp # L. 


Proof: By induction on the length of the execution. The base case consists of proving that 
the invariant is true in the initial state. Fix p,q and v. In the initial state we have that 
rcvd-rgst|q, v.id], = L. Hence the invariant is vacuously true. 

For the inductive step assume the invariant is true in s. We need to prove that it is true in s’ 
for any possible step (s,7,s’). Fix p,q,v. We prove the invariant considering each possible action 
nm. Assume that s’.rcud-rgst{q, v.id|p A L. 


1. 7 = VS-NEWVIEW(v)p. 


Since s’.curp = v we have that s’.cury # L (vs cannot deliver L, it is not a view). 


2. 7 = vs-GPRCV(( “registered”)) p,q- 
By the precondition of 7 (see VS) we have that s.current-viewid[p| 4 L. By Invariant 4.2 we 
have s.cur.idy = s.current-viewid|p] # -L. Hence s’.cur.idy = s.cur.idp # L. 

3. Other actions. 


Variables rcvd-rgst(q, v.id], and curp are not modified. Hence the assertion cannot be made 
false. 


Invariant 4.7 (DVS-IMPL) 
In any reachable state, if cur.idy = L then act) = vo and amb, = {}. 
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Proof: By induction on the length of the execution. The base case consists of proving that 
the invariant is true in the initial state. Fix p. In the initial state we have that act, = vo and 
amb,y = {}. 

For the inductive step assume the invariant is true in s. We need to prove that it is true in s’ 
for any possible step (s, 7,8’). Fix p. We prove the invariant considering each possible action 7. 
Assume that s’.curp = L. Since no actions sets curp to L it must be s.curp = L. 


1. m7 = vs-aprev({ “info”,v,V)) pq - 


This cannot happen. Indeed by precondition of a (see vS) we have that s.current-viewid[p| 
L. By Invariant 4.2 we have s.cur.idy = s.vS.current-viewid[p| Hence s!.cur.id) = s.cur.idy # 
. But we know that s’.cur.id = L. 


2. 17 = DVS-NEWVIEW(v)v. 


Cannot happen. Indeed the precondition of 7 says that v = s.curp. Since s.cur.id = L, we 
have v = L. Thus the precondition v.id > client-cur.idy cannot be satisfied (1 cannot be 
strictly greater than any view identifier). 


3. 7 = DVS-GARBAGE-COLLECT(v). 
Cannot happen. Indeed by Invariant 4.6 we have that s.cury # 1. But we know that 
8.CUTyp = 1. 

4. Other actions. 


Variables curp, act) and amb, are not modified. Hence the assertion cannot be made false. 


0 


The following invariant states that if an “info” message is in transit for view v or has been 
received by some process g in view v then there exists a process p that has sent the “info” in view 
v and such that its current view is either v or a later one. 


Invariant 4.8 (DVS-IMPL) 
In any reachable state, let C be the following condition: 


( “info”, x, X) € msgs-to-vs[g|y or (“info”,«,X) € pendinglp,g] or (( “info”,x,X),p) € 
queuel[g] or info-rcvd[p, glq = (x, X). 


If C is true then info-sent|g], = (x, X) and cur.idy, > g. 


Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. Fix p,q,g,z and X. In the initial state msgs-to-vs[g], = A, 
pending|p, g] = A, queue[g] = A and info-rcvd|p, g]q = -L. Hence, in the initial state, C' is false and 
the invariant is vacuously true. 

For the inductive step assume that the invariant is true in a reachable state s. We need to 
prove that it is true in state s’ for any possible step (s, 7, s’) of the execution. Fix p,q,g,x, and 
X and assume that C is true in s’. 
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1. 7 = VS-NEWVIEW(v)p. 


By the code of a, s'.curp = v. Assume v.id # g. Then the code of a shows that none of 
msgs-to-vs|g|p, pending|p, g], queue[g] or info-rcvd|p, g]q is changed during this step. Thus C 
is true also in s. By the inductive hypothesis we have s.info-sent[g]) = (x, X) and cur.id, > g. 
Since we are considering the case v.id # g, we have that info-sent[g], is not changed by z. 
Moreover the precondition of 7 (see VS) shows that s’.current-viewid|p| > s.current-viewid|p]. 
By Invariant 4.2, cur.idyp = current-viewid[p], so s'.cur.idp > s.cur.idy. This completes show- 
ing the conclusion for the situation w.id ¥ g. 


Assume now v.id = g. The code shows s’.cur.id) = g as required. It remains to show that 
(x, X) € info-sent|g]p. 

Action 7 does not alter the values of pending[p,g], queue[g] and info-rcvd[p,g]q and ap- 
pends ( “info”, s.actp,s.ambp) to msgs-to-vs[g]p. We claim that it must be x = s.act, and 
X = s.amby. Indeed if it is not so, then condition C' is true also in state s (for the given 
p.4,9,x,X) and by the inductive hypothesis we have s.cur.idp > g = w.id. By Invariant 4.2, 
s.current-viewid[p| > w.id. But this contradicts the precondition of 7 (see vs). 

Thus x = s.actp and X = s.amb,. Then the code of 7 shows that («, X) € info-sent[g]p, as 
required. 


2. 7 = vs-eprev(( “info”, v,V))p,¢ - 


If g # cur.idy then since C is true in s’ it is true also in s (for the given p,q,g,2,X). Thus 
the inductive hypothesis is true. Since the code does not change info-sent[g|, and cur.id,, the 
invariant follows from the inductive hypothesis. 


Hence assume that g = cur.idg. First consider the case x = v and X = V. In this case, by 
the precondition of m (see vS) we have that (( “info”, x, X),p) € queuelg]. Then the invariant 
follows from the inductive hypothesis. 


Consider now the case x # v or X # V. In this case, by the code, we have that s’.info-rcvd[p, g]q 
(z,X). Since C is true in s’, it must be that ( “info”, 2, X) € msgs-to-vs[g], or ( “info”, «,X) € 
pending|p, g| or (( “info”, x, X),p) € queuelg] is true in s’. Variables msgs-to-vs[g],, pending|p, g| 
and queuelg] are not changed by a. Hence C is true in s. The invariant follows from the in- 
ductive hypothesis. 


3. T = Vs-GPSND(( “info”, v, V))p- 


If g # client-cur.id, then since C is true in s’ it is true also in s (for the given p,q, 9,2, X). 
Thus the inductive hypothesis is true. Since the code does not change info-sent[g], and cur.idp, 
the invariant follows from the inductive hypothesis. 


Hence assume that g = client-cur.idy. First consider the case x = v and X = V. In this case, 
by the precondition of 7 (see DVS-IMPL) we have that (( “info”, x, X),p) € msgs-to-vs|g]. Then 
the invariant follows from the inductive hypothesis. 


Consider now the case x 4 v or X # V. Since C is true in s’ we have that C is true in s too. 
Indeed no ( “info”, z,X) message is deleted and info-rcvd[p, g]q is not changed. The invariant 
follows from the inductive hypothesis. 
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4. m7 = vs-ORDER(( “info”, v,V),p, 9). 


First consider the case x = v and X = V. In this case, by the precondition of 7 we have that 
(( “info”, x, X),p) € pending|g]. Then the invariant follows from the inductive hypothesis. 


Consider now the case x #4 v or X # V. Since C is true in s’ we have that C is true in s too. 
Indeed no ( “info”, z, X) message is deleted and info-rcvd[p, g]q is not changed. The invariant 
follows from the inductive hypothesis. 


5. Other actions. 


Condition C never changes from false to true and variables info-sent|g]) and cur.idy are not 
modified. Hence the assertion cannot be made false. 


0 


The following invariant states that ifa “registered” message for view v has been sent by process 
p then variable reg[v.id], is set to true (that is, the view has been registered by the client at p). 


Invariant 4.9 (DVvS-IMPL) 
In any reachable state, let C be the following condition: 


(“registered”) € msgs-to-vs[g], or (“registered”) € pending|p,g| or (“registered”,p) € 
queue[g] or rcvd-rgst[p, gq = true. 


If C is true then reg|g|p = true. 


Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. Fix p,g,q. In the initial state we have that msgs-to-vus/ GIy = x, 
pending|p, g] = A, queue[g] = and rcvd-rgst|p, g], = false. Hence C is false in the initial state 
and the invariant is vacuously true. 

For the inductive step assume the invariant is true in s. We need to prove that it is true in s’ 
for any possible step (s,7, 8’). Fix p,g,q and assume that C is true in s’. 


1. 7 = DVS-REGISTERy. 


If s.client-cur.idy # g then C is true also in s and the invariant follows from the inductive 
hypothesis. Hence assume s.client-cur.id) = g. By the code of 7 we have that we have 
reg|g|p = true. 


2. 7 = Vs-GPSND({ “registered”)) p. 


If s.current-viewid|p] 4 g then C is true also in s and the invariant follows from the inductive 
hypothesis. Hence assume g = s.current-viewid[p]. By Invariant 4.2 we have that s.cur.idy = 
s.current-viewid|p]. Hence s.cur.id, = g. By the precondition of 7 (see DVS-IMPL) we have 
that (“registered”) € s.msgs-to-vs[g],. Hence C' is true in s and the invariant follows from the 
inductive hypothesis. 


3. WT =VS-ORDER(( “registered”, p’, q’)). 


If p’ # p or g' # g then C is true also in s and the invariant follows from the inductive 
hypothesis. Hence assume p’ = p and g’ = g. By the precondition of 7 we have that 
(“registered”) € s.pending|[p, g]. Hence C is true also in s and the invariant follows from the 
inductive hypothesis. 
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4, 


Other actions. 


Condition C' never changes from false to true and variable reg[g], is not modified. Hence the 
assertion cannot be made false. 


The following invariant states some facts about views in Jot?eg. 


Invariant 4.10 (Dvs-IMPL) 
In any reachable state: 


1. 


act, € TotReg. 


2. If info-sent|g|p = (x, X) then « € TotReg. 


3. usep M TotReg # {}. 


Proof: First notice that Part 3 follows easily from Part 1 and the fact that, by definition, 
act) € usey. Hence we only need to prove Parts 1 and 2. 


By induction on the length of the execution. The base case consists of proving that the 


invariant is true in the initial state. For Part 1, fix p. In the initial state act) = vo and vo is 
totally registered by definition. For Part 2, fix p,g. In the initial state info-sent[g|, = -L. Hence 
the invariant is vacuously true. 


For the inductive step assume the invariant is true in s. We need to prove that it is true in 


s' for any possible step (s, 7,8’). Fix p, g, and X. We prove the invariant by considering each 
possible action. 


1. 


7] = VS-NEWVIEW(v)>p. 
Part 1 is still true in s’ because act, is not modified (as well as Tot?Req). 


Consider Part 2 now. Assume that s’.info-sent|g], = (a, X). If v.id ¢ g then s.info-sent[g|, = 
(z,X) then by the inductive hypothesis we have that x € s.JotReg. Since no view is ever 
removed from TJotReg we have that x € s’.JotReg, as needed. Hence we can further assume 
that v.id = g. Since s’.info-sent[g], = (x, X) and action m sets info-sent[g]) = (actp, ambp) it 
must be that s.actp =x and s.ambyp = X. 


By the inductive hypothesis, Part 1, we have that s.act, € s.JotReg. But x = s.act, and no 
view is removed from TotReg. Hence x € s’.TJotReg. Thus Part 2 is still true in s’. 


. 7 = vs-aprov({ “info”, v,V))p,¢q- 


Consider Part 1 first. If s’.acty = s.actp then Part 1 follows by the inductive hypothesis. Hence 
assume that s’.act) # s.act). By the code we have that s’.act, = v. Thus we have to prove that 
v € TotReg. By the precondition of m (in vs) we have (( “info”,v,V),q) € s.queue[cur.idp]. 
Then Invariant 4.8 implies that s.info-sent[cur.idp|q = (v,V). By the inductive hypothesis, 
Part 2, we have that v € s.JotReg, as needed. 


Part 2 is preserved because info-sent[g|, is not modified. 
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3. 7 = DVS-GARBAGE-COLLECT(v)p. 


Consider Part 1 first. If s'.actp = s.act) then Part 1 follows by the inductive hypothesis. 
Hence assume that s’.act) # s.actp. By the code we have that s’.act) = v. Hence we have to 
prove that v € JotReg. By the precondition of 7 we have that rcvd-rgst|q, v.id] = true for all 
q € v.set. Then Invariant 4.9 implies that v € JotReg. 


Part 2 is preserved because info-sent[g|, is not modified. 


4. Other actions. 


Variables act,, info-sent|g|, (as well as JotReg) are not modified. Hence the assertions cannot 
be made false. 


0 


The following invariant states that if process q is in a view which has been attempted by process 
p (which may or may not be q itself) then the current view of q is either v or a later one. 


Invariant 4.11 (Dvs-IMPL) 
In any reachable state, if v € attempted, and q € v.set then cur.idg > v.td. 


Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. Fix p,v and suppose that v € attempted, and q € v.set. 
If p ¢ Po then attempted, = {}, a contradiction. On the other hand, if p € Po then since 
v € attempted,, it must be that v = vp. Moreover since q € v.set we have that q € Pp. Hence 
curg = Vo, SO cur.idg > v.id, as needed. 

For the inductive step assume the invariant is true in state s. We need to prove that it is true 
in s’ for any possible step (s,7,s’). Fix p and v and assume that v € s’.attempted, and q € v.set. 
We distinguish two cases. 


1. v € s.attempted,. 
By the inductive hypothesis we have that s.cur.idg > v.id. By the monotonicity of cur.id we 
have that s’.cur.idg > s.cur.idg. 

2. v ¢ s.attempted,. 
It must be 7 = pvs-NEWVIEW(v),. We consider two possible cases: gq = p and q # p. 


Assume that q = p. Then Invariant 4.3 implies that s'.client-cur, > v.id. Since s'.cur.idy = 
s'.client-curp, we have that s’.cur.idpy > v.id, as needed. 


Assume that q 4 p. Then the precondition of x says that s.info-rcvd[q, v.id] # L. By Invariant 
4.8 (used with p and q interchanged) we have that cur.idg > v.id, as needed. 


The following invariant states properties of views in the use set. 


Invariant 4.12 (Dvs-IMPL) 
In any reachable state: 
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1. If curp A L and w € usep, then w.id < cur.idp. 


2. If curp # L and client-cur, # cury and w € usep, then w.id < cur-idy. 


3. If info-sent|g|p = (x, X) and w € {x} UX then w.id < g. 
Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. Consider Part 1 first. In the initial state we have that 
use, is either empty or contains only vp. In the former case Part 1 is vacuously true. In the 
latter case we have that w = vo and the invariant follows from the fact that go is the minimum 
element of G. Parts 2 and 3 are vacuously true. Indeed in the initial state client-curp = curp and 
info-sentlg|p = L. 

For the inductive step assume the invariant is true in state s. We need to prove that it is true 
in s’ for any possible step (s, 7, s’). Fix p,g,2,X and w. 

We prove that the invariant is still true in s’ by considering each possible action 7. 


1. 7 = VS-NEWVIEW(v)p 


First consider Part 1. Assume that s’.curp # L and w € s’.usep. Then w € s.usep. If 
$.cury = L, then, by Invariant 4.7, w = vg. Since vp.id is the minimum element of G, we have 
that w.id < s’.cur.idpy. So assume that s.curp # L. In this case, by the inductive hypothesis, 
Part 1, we have that w.id < s.cur.idy, which implies w.id < s'.cur.idp. 

Hence Part 1 is still true in s’. Since we actually proved that w.id < s’.cur.idy also Part 2 is 
still true in s’. 

Now consider Part 3. Assume that s'.info-sent|g]) = (x, X) and w € {x}UX. If g F v.id then 
we have that s.info-sent|g|, = (x, X). By the inductive hypothesis, Part 3, we have w.id < g, 
as needed. Hence assume g = v.id. By the code of 7, we have that s.usep = {a} UX. Now if 
$.cury = L, then by Invariant 4.7, w = vo. Since vg.id is the minimum element of G, we have 
that w.id < v.id = g, as needed. So assume further that s.cur, # L. In this case, the inductive 
hypothesis, Part 1, implies that w.id < s.cur.idp, which implies w.id < s'.cur.idy = v.id = g, 
as needed. 


2. 17 = DVS-NEWVIEW(v)p 
Consider Part 1 first. The only possible new element added to usey is v. Since v = s'.cur.id, 
Part 1 still holds in s’. Part 2 is vacuously true, because s’.client-curp = s'.curp. Part 3 is 
preserved because info-sent|g], is not modified. 

3. T = DVS-GARBAGE-COLLECT(v) p 


Consider Part 1. Assume that s’.curp # L and that w € s’.usep. By the code s’.curp = 8.curp. 
If w € s.usep then by the inductive hypothesis Part 1 is true in s and thus it is still true in s’. 
Hence assume that w ¢ s.usep. By the code, this cannot happen because no view is added to 
USEp. 


Part 2 can be proved in a similar way. Part 3 is preserved because info-sent|g]p is not modified. 


4. 7 = vs-epRcv(( “info”, x, X))q,p 


The proof is exactly as in the previous case. 
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5. Other actions. 


Variables usép, curp, client-curp and info-sent[g]) are not modified. Hence none of the asser- 
tions can be made false. 


0 


The following three invariants, say that certain views appear in use sets, or in “info” messages, 
unless they have been garbage-collected. 


Invariant 4.13 (DvSs-IMPL) 
In any reachable state, if w € attempted, then either w € usepy or w.id < act.idy. 


Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. Fix p,w and suppose that w € attempted,. If p ¢ Po then 
attempted, = {}, a contradiction. On the other hand, if p € Po then since w € attempted, it 
must be that w = vo. But in this case also acty = vg, 80 Vp € Usep, as needed. 

For the inductive step assume the invariant is true in state s. We need to prove that it is true 
in s’ for any possible step (s,7,s’). So fix w and p such that w € s'.attempted,. We distinguish 
two possible cases. 


1. w € s.attempted,. 


By the inductive hypothesis we have that either w € s.usey or w.id < s.act.idp. In the 
latter case, because of the monotonicity of act.id,, we have w.id < s’.act.idy. So assume that 
w € S.usey. If w € s’.usey we are done, so assume further that w ¢ s'.usep. Then it must be 
that either 7 = Dvs-GARBAGE-COLLECT(v)p, OF 7 = vs-GPRCV(( “info”, x, X))>» for some r. In either 
case, the code implies that s’.actp > w.id. 


2. w ¢ s.attempted,. 


It must be 7 = pvs-Newview(v)p. By the code, view v is inserted into attempted,, but also into 
amb, (and hence into usep). Thus the invariant is still true in s’. 


Invariant 4.14 (Dvs-IMPL) 
In any reachable state, if info-rcvd|q,g\p = (x,X) and w € {a} UX, then either w € usep or 
w.id < act.idy. 


Proof: By induction on the length of an execution. The base case consists of proving that the 
invariant is true in the initial state. In the initial state info-rcvd|q, g]p = -L for any p,q, g. Hence 
the statement is vacuously true. 

For the inductive step assume the invariant is true in state s. We need to prove that it is true 
in s’ for any possible step (s, 7, s’). Fix p, q, g, x, X and w, and assume that s’.info-rcvd|q, g|p = 
(x, X), and w € {x} UX. We consider two cases: 


1. s.info-rcvd|q, glp = (x, X) 


By the statement applied to s, we obtain that either w € s.usep, or s.act.idy > w.id. In the 
latter case, s’.act.id, > w.id, because of monotonicity of act.idp. So assume that w € s.usep. 
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If w € s’.usep then we are done, so assume further that w ¢ s’.usep. (That is, w is garbage- 
collected.) 

Then it must be that either 7 =pvs-GARBAGE-COLLECT(v)p) OF 7 = vs-GPRCV(( “info”, x, X))»» for 
some r. In either case, the code implies that s’.act) > w.id. 


2. s.info-rcvd|q, g]p A (x, X) 


Then m7 = vs-aprev((“info”,2,X))qp. If w € s’.usey then we are done. Hence assume that 
w ¢ s'.usep. By the code, we have that s’.act, > w.id (that is, w is garbage-collected). 


Invariant 4.15 (DvS-IMPL) 
In any reachable state, if info-sentlg]) = (x,X), w € attempted,, and w.id < g, then either 
wé{x}UX or w.id < x.id. 


Proof: By induction on the length of an execution. The base case consists of proving that the 
invariant is true in the initial state. In the initial state, info-sent[g], = 1 for all g,p, so the 
statement is vacuously true. 

For the inductive step assume the invariant is true in state s. We need to prove that it is true in 
s’ for any possible step (s, 7, s’). Fix p, g, w, z, and X, and assume that s’.info-sent[g]p = (x, X), 
we s'.attempted,, and w.id < g. We consider four cases: 


1. s.info-sent|g]p = (x, X) and w € s.attempted,. 
Then the statement for s implies that either w € {x} UX or w.id < x.id. In either case the 
statement is true in s’ also. 

2. s.info-sent|g]p # (x, X) and w ¢ s.attempted,. 


This cannot happen because both conditions cannot become true in a single step: the first only 
becomes true by means of a vs-NEWVIEW(v)», for some view v, while the second only becomes 
true by means of pvs-NEWVIEW(w)p. 


3. s.info-sent|g]p # (x, X) and w € s.attempted,. 


It must be m = vs-NEWvIEW(v)p, for some v, x must be s.act,, and X must be s.ambp. 
Invariant 4.13 implies that either w € s.usep or w.id < s.act.idp. Now, s.usep = {8.acty}U 
s.ambp = {x} UX. So we have that either w € {xz} UX or w.id < x.id, as needed. 


4. s.info-sent|g|p = (x, X) and w ¢ s.attempted,. 


Then 7 must be pvs-Newvinw(w),. We claim that this cannot happen: Since s.info-sent|g]p = 
(x, X), by Invariant 4.4 we have s.cur.id, > g. Since g > w.id, we have s.cur, > w.id. But 
the precondition of 7 requires that s.cury = w.id. Hence 7 is not enabled in state s. 
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Invariant 4.16 says that two attempted views having no intervening totally registered view, 
and having a common member, q, that has attempted the first view, must intersect in a majority 
of processors. This is because, under these circumstances, information must flow from q to any 
processor that attempts the second view. 


Invariant 4.16 (DvS-IMPL) 
In any reachable state, suppose that v € attempted,, q € v.set, w € attempted,, w.id < v.id, and 
there is no x € TotReg such that w.id < x.id < v.id. Then |v.set Nw.set| > |w.set|/2. 


Proof: By induction on the length of an execution. The base case consists of proving that the 
invariant is true in the initial state. In the initial state, only vo is attempted, so the hypotheses 
cannot be satisfied. Thus, the statement is vacuously true. 

For the inductive step assume the invariant is true in state s. We need to prove that it is true in 
s’ for any possible step (s,7, 8’). Fix v, w, p, and q, and assume that v € s'.attempted,, q € v.set, 
we s'.attempted,, w.id < v.id, and there is no x € s'.JotReg such that w.id < xvid < v.id. Then 
also there is no x € s.TotReg such that w.id < x.id < v.id. We consider four cases: 


1. v € s.attempted, and w € s.attemptedg. 


Then the statement for s implies that |v.set N w.set| > |w.set|/2, as needed. 


2. v ¢ s.attempted, and w ¢ s.attempted,. 
This cannot happen because we cannot have both v and w becoming attempted in a single 
step. 

3. ug s.attempted, and w € s.attempted,. 


Then 7 must be pvs-NEWVIEW(v),. Since q € v.set, by the precondition of a we have that 


s.info-rcvd|q, v.id]y = (x, X) for some x and X. Then Invariant 4.8 implies that s.info-sent[v.id]q = 


(x, X). Then (since w.id < v.id), Invariant 4.15 implies that either w € {x}UX or w.id < a.id. 
If wid < x.id, then we obtain a contradiction. Indeed by Invariant 4.10 x € s.TJotReg and by 
Invariant 4.12, Part 3 (used with w = x) we have x.id < v.id. This contradicts the hypothesis. 
Sow € {a} UX. 

Now by Invariant 4.14 we have that either w € s.usep or w.id < s.act.id. In the former case, 
by the precondition of 7, we have |v.set N w.set| > |w.set|/2. In the latter case, we obtain a 
contradiction. Indeed by Invariant 4.10 we have s.act, € TotReg. Moreover by the precondi- 
tion of 7, s.curp cannot be L and s.curp > s.client-cury and, by definition, s.actp € s.useép. 
Hence by Invariant 4.12, Part 2, we have s.act.idy < s.curp = v.id. 


4. v € s.attempted, and w ¢ s.attemptedg. 


Then 7 must be pvs-Newvimw(w),- But this cannot happen. Indeed since v € s.attempted, 
and q € v.set, Invariant 4.11 implies that s.cur.idy > v.id. Since v.id > w.id, we have 
s.cur.idg > w.id. But the precondition of action m requires s.cur.idg = w.id, so m is not 
enabled in s. 
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Invariant 4.17 says that any attempted view v intersects the latest preceding totally registered 
view w in a majority of members of w. 


Invariant 4.17 (Dvs-IMPL) 
In any reachable state, suppose that v € Att, and w € TotReg, w.id < v.id, and there is no 
x € JotReg such that w.id < x.id < v.id. Then |v.set N w.set| > |w.set|/2. 


Proof: By induction on the length of an execution. The base case consists of proving that the 
invariant is true in the initial state. In the initial state, only vp is attempted, so the hypotheses 
cannot be satisfied. Thus, the statement is vacuously true. 

For the inductive step assume the invariant is true in state s. We need to prove that it is 
true in s’ for any possible step (s, 7,5’). Fix v and w, and assume that v € s’.Att, w € s’.TotReg, 
w.id < v.id, and there is no x € s’.JotReg such that w.id < x.id < v.id. We consider four cases: 


1. v € s.Att and w € s.TotReg. 


Then, from the inductive hypothesis we have |v.set N w.set| > |w.set|/2. 


2. vg s.Att and w ¢ s.TotReg. 


This cannot happen because we cannot have both v becoming attempted and w becoming 
totally registered in a single step. 


3. v € s.Att and w € s.TotReg. 


Then 7 must be pvs-NEWvIEW(v), for some p. The precondition of 7 implies that, for any view 
y € S.usep, |v.set N y.set| > |y.set|/2. Hence to prove the claim it is enough to prove that 
w € 8.USep. We proceed by contradiction assuming that w ¢ s.usep. 


By Invariant 4.10, Part 3, s.use,  s.JotReg # {}. Let m be the view in s.use, M s.TotReg 
having the biggest identifier. We know that m 4 w because w ¢ s.usep. Also, m # v, because 
m € s.JotReg and v ¢ s.TotReg. It follows that m.id F v.id. 


We claim that m.id < w.id. We have already shown that m.id 4 w.id. Suppose for the sake 
of contradiction that m.id > w.id. From the precondition of action 7 we have that s.cur = v 
and hence s.cur # L. Also from the precondition of a we have that s.client-cur, < s.curp. 
Since m € s.useéy, Invariant 4.12, Part 2, implies that m.id < s.cur.idp and since s.cur = v we 
have we have m.id < v.id. So w.id < m.id < v.id. Since m € s’.TotReg, this contradicts the 
hypothesis of the inductive step. Therefore, m.id < w.id. 


Let n be the view in s.JotReg that has the smallest id strictly greater than that of m. 
Remember that w € s’.JotReg and since 7 =pvs-NEWVIEW(v)» we have that w € s.JotReg; 
thus n exists and it holds m.id < n.id < w.id < v.id. Since m € s.useép, the precondi- 
tion of implies that |v.set M m.set| > |m.set|/2. By the statement applied to state s, 
|n.set 1 m.set| > |m.set|/2. Hence there exists a processor g € v.setM n.set. By the pre- 
condition of 1, s.info-rcvd[q, v.id], = («,X) for some «,X. Then Invariant 4.8 implies that 
s.info-sent[v.id], = (a,X). Then Invariant 4.12, Part 3 (used with w = «), implies that 
z.id < vid. Since n € s.TJotReg, we have that n € s.attempted,. Then Invariant 4.15 (used 
with w = n) implies that either n € {x} UX or n.id < x.id. In either case, {x} UX contains 
a view y € s.JotReg (either n or x) such that nid < y.id < v.id. Then Invariant 4.14 implies 
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that either y € s.usep or y.id < s.act.idp. By Invariant 4.10, Part 1, s.act) € s.TotReg and by 
definition, s.act) € s.use,. So in either case, the hypothesis that m is the totally registered 
view with the largest id belonging to s.use, is contradicted. 


4. v€ s.Att and w ¢ s.TotReg. 


Then + must be pvs-REGISTER, for some p. Let m be the view in s.TotReg with the largest id 
that is strictly less than w.id. By the statement for s, we know that |w.setMm.set| > |m.set|/2 
and |v.setM m.set| > |m.set|/2. Hence there is a processor qg € w.setM v.set. 


Since v € s.Att, there exists a processor r such that v € s.attempted,. Thus also v € 
s'.attempted,. Since w € s'.JotReg, we have that w € s'.attempted,. By assumption, there 
is no view x € s’.JotReg such that w.id < x.id < v.id. By Invariant 4.16 applied to state s’ 
(with p =r), we have that |v.setM w.set| > |w.set|/2, as needed. 


O 


The final invariant, a corollary to Invariant 4.17, is instrumental in proving that Dvs-IMPL 
implements DVS. 


Invariant 4.18 (DvS-IMPL) 
In any reachable state, if v,w € Att, wid < vid, and there is no x € TotReg with w.id < xid < 
v.id, then v.set Nw.set # {}. 


Proof: Suppose that v and w are as given. We consider two cases. 


1. w € TotReg. 


Since there is no x € JotReg, Invariant 4.17 implies that |v.set N w.set| > |w.set|/2, which 
implies that v.set MN w.set 4 {}, as needed. 


2. w ¢ TotReg. 

Then let Y = {yly € TotReg, y.id < w.id}. We first show that Y is nonempty: Invariant 4.5 
implies that vg € JotReg and that vo.id < w.id. If vg.1d = w.id, then by Invariant 4.1, we have 
w= vg. But then w € JotReg, a contradiction to the definition of this case. So we must have 
ug.id < w.id, which implies that v9 € Y, so Y is nonempty. 

Now fix z to be the view in Y with the largest id. We have that there is no x € JotReg 
with z.id < x.id < vid. Then Invariant 4.17 implies that |w.set M z.set| > |z.set|/2 and 
|v.set N z.set| > |z.set|/2. Together, these two facts imply that v.set MN w.set # {}, as needed. 


O 


4.3.2 The abstraction function 


We prove that DvVS-IMPL implements Dvs by defining a function F that maps states of DVS-IMPL 
to states of Dvs and proving that this function is a abstraction function. Section 4.3.2.1 contains 
the definition of the function F along with auxiliary invariants, then Section 4.3.2.2 gives the 
proof that F is an abstraction function. 
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4.8.2.1 Definition of F 


DVS-IMPL uses VS to send client messages and messages generated by the implementation ( “info” 
and “registered” messages). The abstraction function discards the non-client messages. Thus, if ¢ 
is a finite sequence of client and non-client messages, we define purge(q) to be the queue obtained 
by deleting any “info” or “registered” messages from q, and purgesize(q) to be the number of 
“info” and “registered” messages in q. Figure 5 defines the abstraction function F. 


Let s be a state of DVS-IMPL. The state t = F(s) of DvS is the following. 
e t.created = Upeps.attempted, 


e for each p € P, t.current-viewid[p| = s.client-cur.idp 


e for each g € G, t.attempted|g] = {p|g = v.id,v € s.attempted, } 


e for each g € G, t.registered|g] = {p|s.reg[g]p } 
for each p € P, g € G, t.pending|p, g| = purge(s.pending|p, g])+purge(s.msgs-to-vs|g]p) 
for each g € G, t.queue[g] = purge(s.queue[g]) 


for each pE PG EG, 
t.neat[p, g] = s.neztlp, g] — purgesize(s.queue[g](1..next[p, g] — 1)) — |s.msgs-from-vs|g]p| 


for each pE PG EG, 
t.next-safelp, g] = 
s.next-safelp, g] — purgesize(s.queuelg](1..next-safe[p, g] — 1)) — |s.safe-from-vs|g]p| 


Figure 5: The abstraction function F. 


Next we give some simple consequences of the definition of 7. They deal with the messages 
delivered by Dvs-IMPL. They state that these messages are exactly the ones that DVS would 
deliver to the client. 


Invariant 4.19 (DvS-IMPL) 
In any reachable state s, if s.msgs-from-vs[g]p = ((m1, 91); (M2, 92), +; (ME, dk)), then we have 
that F (s).queuelg](nezt| ,g].-neatl 9] = 1) oa ((m1, 1); (m2, 42); a (Mk k))+ 


Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. In the initial state no message is in msgs-from-vs|g]p. Hence 
the invariant is vacuously true. 

For the inductive step, assume that the invariant is true in state s. We need to prove that it is 
true in state s’ for any possible step (s,7,s'). Fix p,g and m1, q1,™2,qQ,---; Mk: Qk and assume 
that s’.msgs-from-vs[g]p = ((m1, 1), (M2, G2), ++, (Mz, d)). We distinguish the following cases. 


1. s.msgs-from-vs|g|p = ((1, G1) 5 ++) (Mk=1; Uk—-1))- 
It must be 7 =vs-Gprcv(ms)q,,p- By the inductive hypothesis we have that 
F (s).queue[g](nezt| ,g]--neat 9] sia, ia 2) = ((m1, 1); oo) pea, Gk=1)): 
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By the code in vs we have that nezt[p,g] is increased by one and by the code in DvS we 
have that the size of msgs-from-vs[g]p also increases by one. Hence by the definition of F, we 
have that F(s’).nezt|p,g] = F(s).next[p, g]. Moreover F(s’).queue[g] = F(s).queuelg] and by 
the precondition of 7 we have that F(s).queue[g](s.nezt[p, g] + k — 1) = (mz, q,). Thus the 
invariant is still true in s’. 


2. s.msgs-from-vs|g|p = ((m, q); (m1, M1); (ma, q2), ae) (mz, dk))- 
Then a =pvs-Gprcv(m)q,»- By the inductive hypothesis we have that 
F (s).queuelg](nect| ,g].-neatl 9] + k) = ((m, q); (m1, M1): (ma, q2), sine (mg-1, dk-1))- 


By the code we have that nezt[p, g] is incremented by one. Since F(s’).queue[g] = F(s).queuelg], 
the invariant is still true in s’. 


3. s.msgs-from-vs|g]p = s'.msgs-from-vs|g]p 


By the inductive hypothesis the assertion is true in state s. For any possible action in this case 
F(s').nezt|p, g] = F(s).nezt[p,g| and the portion of F(s).queue[g] involved in the statement 
of the invariant never changes because messages are only appended to quewe[g]. Thus the 
assertion cannot be made false. 


4. Other cases. 


Not possible. Indeed msgs-from-vs[g]p either stay the same or is changed by appending a 
message or deleting the head. 


O 
The following invariant follows easily from the previous one. It just states that the next message 
delivered by DVS-IMPL to a processor p is the same one that Dvs delivers. 
Invariant 4.20 (DVS-IMPL) 
In any reachable state s, if (m,q) is head of s.msgs-from-vs|g]p, then F(s).queue[g](neat[p, g]) = 
(m,q): 
Proof: Follows easily from previous one. O 
Similar invariants hold for the delivery of safe messages. 
Invariant 4.21 (DVS-IMPL) 
In any reachable state s, we have that if s.safe-from-vs|g]y = ((m1, q1), (M2, G2), +, (ME, e)), then 
F (s).queuelg](nezt-safelp, 9]; next-safelp, 9] ke 1) i ((m1, M1); (ma, q2), Bore) (mz, dk))- 
Proof: The proof is as for msgs except that it uses the safe-from-vs queue instead of msgs-from-vs 
and the pointer nezt-safe instead of nezt. o 


Invariant 4.22 (DvS-IMPL) 
In any reachable state s, if (m,q) is head of s.safe-from-vs|g|p, then F(s).queue[g|(next-safelp, g]) = 


(m,q): 
Proof: Follows easily from previous one. O 


Notice that v is totally registered in state s of DVS-IMPL if and only if it is totally registered in 
the state of Dvs that appears in state F(s) of Dvs. 
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4.8.2.2 Proof that F is an abstraction function 


In order to prove that F is an abstraction function we need to prove that (a) for any initial state 
s of DVS-IMPL we have that F(s) is an initial state of Dvs, and that (b) for any possible step 7 
of DVS-IMPL there exists an execution fragment a of DVS such that the trace of a is equal to the 
trace of 7, that is, w and 7 have identical externally observable behaviors. Lemmas 4.23 and 4.24 
prove this. 


Lemma 4.23 If s is an initial state of DVS-IMPL then F(s) is an initial state of DVS. 


Proof: Let sg be the unique initial state of DVS-IMPL and tp the unique initial state of Dvs. 

We have so.attempted, = {vo} for p € Po and so.attempted, = {} for p ¢ Po. By the definition 
of F and the fact that Py # {} (because all membership sets are defined to be nonempty), we 
have F(s9).created = {vo}. This is as in to. 

We have so.client-curp = {vo} for p € Po and so.client-cur, = 1 for p ¢ Po. By the definition 
of F we have F(sq).current-viewid|p] = go for p € Pp and F(so).current-viewid|p| = L for p ¢ Po. 
This is as in to. 

We have s9.attempted, = {vo} for p € Po and s9.attempted, = {} for p ¢ Po. By the definition 
of F we have F(sq).attempted|go| = Py and F(so).attempted|g] = {} for g go. This is as in fo. 

Let g € G. We have that so.reg[g|p is true if and only if p € Po and g = go. By the definition 
of F we have F(s).registered|go| = Py and F(so).registered[g| = {} for g # go, as in to. 

Let p € P. We have that s9.msgs-to-vs[g]) = A and so.pending|p, g| = A. By the definition of 
F we have F(sq).pending|p, g| = X, as in to. 

Let g € G. We have s9.queuelg] = A. By the definition of F we have F(so).queuelg] = 4, 
as in fo. 

Let p € P,g € G. We have sq.neat[p, g| = 1, purgesize(s.vS.queue[g]) = 0 and so.msgs-from-vs[g]p = 
A. By the definition of F we have F(so).nezt[p,g] = 1, as in to. A similar argument holds for 
nezt-safe. 

Thus F(so9) = to, as needed. qo 


Lemma 4.24 Lets be a reachable state of DVS-IMPL, F(s) a reachable state of DVS, and (s, 7,8’) 
a step of DVS-IMPL. Then there is an execution fragment a of DVS that goes from F(s) to F(s‘), 
such that trace(a) = trace(n). 


Proof: By case analysis based on the type of the action 7. (The only interesting case is where 
T = DVS-NEWVIEW(v)p.) Define t = F(s) and t’ = F(s’). 
1. 7 = VS-CREATEVIEW(v) 
Then trace((s,7,s’)) = A. Action 7 modifies created. The definition of F is not sensitive to 
this change. Therefore, ¢t = t’, and we set a = t. 
2. 1 = VS-NEWVIEW(v)»p 


Then trace((s,7,s’)) = ». Action m modifies cur, info-sent[cur.id]), and current-viewid[p], 
and adds an “info” message to msgs-to-vs[cur.id],. The definition of F is not sensitive to any 
of these changes. Therefore, t = t’, and we set a = t. 
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3. T = VS-GPSND(m)>p 


Then trace((s,,s’)) = A. Action m just moves a message from the queue msgs-to-vs[cur. id], 
to the queue pending[p, current-viewid[p||. The definition of F is not sensitive to this change. 
Therefore, ¢ = t’, and we set a =t. 


4. 17 = VS-ORDER(m, p, 9) 


Then trace((s,7,s’)) = A. Action 7 moves a message from pending[p, g] to queuelg]. We 
consider two cases. 


(a) mE M, 
Then we set a@ = (t¢, pvs-orDER(m,p,g),t’). We claim that pvs-orpDER(m,p,g) is enabled in t: 
Since vs-oRDER(m, p,g) is enabled in s, it follows that m is the head of s.pending|p, g]. By 
the definition of F, m is also the head of t.pending|p, g]. It follows that pvs-orpER(m, p, 9) 
is enabled in f. 
By definition of F, t’ differs from ¢ only in the fact that m is moved from pending|p, g] to 
queue[g]. This is the effect achieved by applying pvs-orpER(m, p, g) to t. 


(b) me~ Me 
Then the definition of F is not sensitive to this change. Therefore, t = t’, and we set 
a=t. 


5. 7 = vs-Gprev(( “info”, v, 8))q,p 


Then trace((s,7,s’)) = ». This action can modify info-rcvd|cur.idp, q|p, act) and amb, (see 
code of Dvs) and causes nezt|p, cur.idp] to be incremented (see code of vs). The definition of F 
is not sensitive to these changes. (The only interesting case is the definition of t.nezt[p, cur.idp], 
where the absolute values of the first two terms on the right-hand side are both increased by 
1, but they cancel each other out.) Therefore, ¢ = t’, and we set a = t. 


6. 7 = vs-GPRCV( “registered” 
g P 


Then trace ((s, 7, 8’) = X. This action can modify rcvd-rgst[cur.id, q|p. It also causes the pointer 
neat[p, cur.idp| to be incremented. The definition of F is not sensitive to these changes. (The 
only interesting case is the definition of t.next[p, cur.id)], where the absolute values of the first 
two terms on the right-hand side are both increased by 1, but they cancel each other out.) 
Therefore, t = t', and we set a = t. 


7. 7 = vs-GPRcV(m)p, m € M, 


Then trace((s,7,s’)) = A. This action copies a message from the sequence queue[cur.id], to 
the sequence msgs-from-vs[p, client-cur|p]|, and causes nezt[p, cur.id)] to be incremented. The 
definition of F is not sensitive to these changes. (The only interesting case is the definition of 
t.neat[p, cur.idp], where the absolute values of the first and third terms on the right-hand side 
are both increased by 1, but they cancel each other out.) Therefore, t = t’, and we set a = t. 


8. 1 = vs-SAFE((m, v, 8))q,p, Mm € { “info”, “registered”} 
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10. 


Then trace((s,7,s’)) = A. Action m just causes nezt-safelp, cur.idp] to be incremented. The 
definition of F is not sensitive to this change. (The only interesting case is the definition of 
t.neat-safe[p, cur.idp|, where the absolute values of the first two terms on the right-hand side 
are both increased by 1, but they cancel each other out.) Therefore, t = t’, and we set a = t. 


. 1% = VS-SAFE(m)>, m € M, 


Then trace((s, 7, s’)) = ». Action 7 adds a message to safe-from-vs[cur.id]) and causes the 
pointer neat-safe[p, cur.idp] to be incremented. The definition of F is not sensitive to these 
changes. (The only interesting case is the definition of t.nezt-safelp, cur.idp], where the abso- 
lute values of the first and third terms on the right-hand side are both increased by 1, but 
they cancel each other out.) Therefore, t = t’, and we set a = t. 


7] = DVS-NEWVIEW(v)p 


Then trace((s,7,s)) =. In DVS-IMPL, this action modifies only variables amb,, attempted, 
client-cury. We have s'.client-curp = v and s’.attempted, = s.attempted, U {v}. By definition 
of F, we have that t’.current-viewid[p] = s’.client-cur.id) = v.id, t'.created = t.created U {v} 
and t'.attempted|v.id] = t.attempted[v.id] U {p}, while all other state variables in t’ are as in t. 


We consider two cases: 


(a) v € t.created. 
In this case, we set a = (t,7’,t’), where 7’ = pvs-NEwvIEW(v)p. The code shows that 7’ 
brings DVS from state ¢t to state t’. It remains to prove that 7’ is enabled in state t, that 
is, that v € t.created and v.id > t.current-viewid[p|. The first of these two conditions is 
true because of the defining condition for this case. The second condition follows from 
the precondition of 7 in DVS-IMPL: this precondition implies that v.id > s.client-cur.idp, 
and by the definition of F we have t.current-viewid[p] = s.client-cur-idp. 


(b) uv ¢ t.created. 
In this case we set a = (t, 7’, t”, 1”, t’), where 7’ = pvs-cREATEVIEW(v)p, 7” = DVS-NEWVIEW(v)p, 
and ¢” is the unique state that arises by running the effect of 7’ from t. The code shows 
that a brings Dvs from state t to state t’. It remains to prove that 7’ is enabled in t and 
that 2” is enabled in t”. 
The precondition of zr’ requires that (i) Vw € t.created, v.id # w.id and (ii) Vw € t.created, 
either dz € s.JotReg satisfying w.id < xid < vid or vid < x.id < w.id, or else v.set N 
w.set # {}. 
To see requirement (i), suppose for the sake of contradiction that w € t.created and 
w.id = v.id. The precondition of 7 in DVS-IMPL implies that v = s.curpy, which implies 
that v € s.created. Since w € t.created, the definition of F implies that w € s.attempted, 
for some q. This implies that w € s.created. But then Invariant 4.1 implies that v = w. 
But this contradicts that fact that v ¢ t.created and w € t.created. 
To see requirement (ii), suppose that w € t.created and there is no x € s.JotReg satisfy- 
ing wid < gid < v.id or vid < x.id < w.id. Since w € t.created, by definition of F, 
w € s.attempted, for some q. Clearly, w € s'.attempted,. Therefore, w € s’.Att. By the 
code of a we have that v € s’ .attempted,. Therefore we also have v € s’. Att. Moreover, 
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11. 


12. 


13. 


14. 


15. 


there is no x € s'.JotReg satisfying w.id < aid < v.id or vid < anid < wid. Then 
Invariant 4.18 implies that v.setM w.set £ {}, as needed to prove that 7’ is enabled in t. 


We now prove that 2” is enabled in state ¢”. The precondition of 2” requires that 
v € t’ created and v.id > t”.current-viewid|p]. The first condition is true because v 
is added to created by x’. The second condition follows from the precondition of 7 in 
DVS-IMPL: The precondition of 7 implies that v.id > s.client-cur.idy. The definition of 
F implies that t.current-viewid|[p] = s.client-cur.id). Moreover, t”.current-viewid|[p| = 
t.current-viewid[p]. It follows that v.id > t”.current-viewid|[p]. Thus 2” is enabled in state 
i 

1 =DVS-REGISTERp 

Then trace((s,7,s’)) = a. Let g be s.client-cur.id), which equals t.current-viewid|p] by the 

abstraction function. If g = 1, then 7 has no effect in DVS-IMPL, so s = s’; thus t = t’, as 

required to show that a brings Dvs from t to t’. Otherwise, g 4 1, so by the code in Dvs- 

IMPL, this action sets reg[g], to true and inserts a “registered” message into msgs-to-vs[g]p. 


By definition of F, t’ is the same as ¢ except that t’.registered|g| = t.registered[g| U {p}. We 
set a = (t, DVS-REGISTER», t’). It is easy to check that pvs-recisTeR, brings DVS from ft to t’. 


7] =DVS-GARBAGECOLLECT(v)p 

Then trace((s,7,s’)) = A. This action can modify act, and amby. The definition of F is not 
sensitive to these changes. Therefore, t = t’, and we set a = t. 

7] =DVS-GPSND(m)>p 

Then trace((s,7,s’)) = 7. We set a = (t, bvs-GPsnD(m)p, t’). We consider two cases: 

(a) s.client-cur.id = L 


Then s = s’. In this case, the definition of F implies that also t.current-viewid|p] = L, 
which implies that the action also has no effect in ¢, which suffices. 

(b) s.client-cur.id A L 
In this case, the action appends m to msgs-to-vs[g]p, where g = client-cur.id). Hence 
we have that s’.msgs-to-vs|g] = s.msgs-to-vs[g|+m. By the definition of F we get that 
t! pending|p, g| = t.pending|p, g|+m. This is the effect of the action in t (using the fact 
that t.current-viewid[p| 4 L.) 


7] = DVS-GPRCV(m)p 


Then trace((s,7,s’)) = a. This action removes the head of msgs-from-vs|g|p, where g = 
cur.idy. We have that s.msgs-from-vs|g]p = m+s'.msgs-from-vs[g|p. Thus t’.next[p,g] = 
t.neat[p,g] + 1. We set a = (t,pvs-cprcv(m),,t’). It is easy to check that the step has the 
required effect in Dvs. The fact that pvs-cprcv(m), is enabled in t follows from Invariant 4.20. 


7] = DVS-SAFE(m)p 


Then trace(z) = 7. This action removes the head of the safe-from-vs[g]), where g = cur.idp. 


We have that s.safe-from-vs[g]) = m+s'.safe-from-vs|g]p. Thus t’.next-safe[p, g] = t.next-safelp, g]+ 
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1. We set a = (t, pvs-Gprcv(m),, t’). It is easy to check that the step has the required effect in 
DVS. The fact that pvs-Gprcv(m), is enabled in ¢ follows from Invariant 4.22. 


Lemmas 4.23 and 4.24 prove that F is an abstraction function from DVS-IMPL to DVS and thus 
the following theorem holds (this is a standard inference, cf. [29]). 


Theorem 4.25 Every trace of DVS-IMPL is a trace of DVS. 


5 An application of Dvs 


Now we demonstrate the utility of DvS by showing how to use it to implement a totally ordered 
broadcast service, called TO, originally defined in [17]. This service accepts messages from clients 
and delivers them to all clients according to the same total order. This kind of service is used as a 
building block for many fault-tolerant distributed applications, e.g., in implementing sequentially- 
consistent shared memory and atomic shared memory. The TO specification is reproduced in 
Figure 6. 


Signature: 

Input: BCAST(a)p, a€ A, pe P : 

Internal: TO-ORDER(a,p),a€A,peEP Outputs “BREVialya Gt Dg GP 
State: 

queue € segof(A x P), init » foreach p€ P:  pending|[p] € segof(A), init » 

nezt[p] € N*°, init 1 

Transitions: 

input BCAST(a)p output BRCV(a)p,¢ 

Eff: append a to pending|[p] Pre: queue(nezt[q]) = (a, p) 


Eff: nezt[g] := nezt[q] +1 
internal TO-ORDER(a, p) 
Pre: a is head of pending|[p| 
Eff: remove head of pending[p] 
append (a, p) to queue 


Figure 6: The TO service 


5.1 The implementation TO-IMPL 


We provide an implementation of TO using DVS as a building block. The implementation is similar 
to the TO implementation provided in [17]. Both algorithms rely on primary views to establish a 
total order of client messages. The difference is that the algorithm in [17] uses a static notion of 
primary and our new algorithm uses a dynamic notion. The algorithm of [17] is built upon the 
vs service, defined in the same paper, that reports non-primary as well as primary views; that 
algorithm uses a simple local test to determine if the view is primary, namely it checks whether the 
view contains a majority of the processes; the algorithm also does some non-critical background 
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work (gossiping information) in non-primary views. In contrast, the algorithm we present here is 
built upon the DvS service, which only reports primary views. Thus the new algorithm is simpler 
in that it does not perform the local tests and does not carry out any processing in non-primary 
views. On the other hand, in the new algorithm, the application programs must perform pvs- 
REGISTER actions to tell the DVS service that they have obtained whatever information they need 
to proceed with regular computation in the new view. The corresponding notion in [17] is that of 
an established view: an established view in the algorithm of [17] corresponds to a registered view 
in our algorithm. Although the new algorithm appears very similar to the one of [17], the fact 
that the DVS service provides more complicated guarantees than the vs service makes the new 
algorithm harder to prove correct. 


The TO-IMPL algorithm involves normal and recovery activity. Normal activity occurs while 
a group view is not changing. Recovery activity begins when a new primary view is presented 
by DvSs, and continues while the members combine information from their previous history, to 
provide a consistent basis for ongoing normal activity. 

During normal activity, each client message received by TO-IMPL is given a system-wide unique 
label, which consists of a view identifier (the one of the view in which the message is received), a 
sequence number and the identifier of the process receiving the message. The association between 
client messages and their unique labels is recorded in a relation content and communicated to 
other processes in the same view using DvS. When a message is received, the label is given an 
order, a tentative position in the system-wide total order the service is to provide. When client 
messages have been reported as delivered to all the members of the view, by the “safe” notification 
of Dvs, the label and its order may become confirmed. The messages associated with confirmed 
labels may be released to the clients in the given order. 

The consistent sequence of message delivery within each view keeps this tentative order con- 
sistent at members of a given view, but it may be not consistent between processes in different 
views. To avoid inconsistencies processes need state exchange at the beginning of a new view. 

When a new primary view is reported by DVS, recovery activity occurs to integrate the knowl- 
edge of different members. First, each member of a new view sends a message, using DVS, that 
contains a summary of that node’s state. The summary of a node’s state contains the following 
information: the association of labels with client messages, stored in content, the order of client 
messages to be reported to the clients, stored in order, a pointer to the next client message to 
be confirmed, stored in nextconfirm and the view identifier of the primary view with the highest 
view identifier in which the order sequence has been modified (stored in highprimary). 

Once a node has received all members’ state summaries, it processes the information in one 
atomic step, 1e., it registers the new view using the pvs-rEecisTER action. The node processes 
state information as follows: it defines its confirmed labels to be the longest prefix of confirmed 
labels known in any of the summaries; it determines the representatives as the members whose 
summary include the greatest highprimary value; adopts as its new order the order of a “chosen” 
representative (the chosen representative is arbitrary but must be the same for all processes) 
extended with all other labels appearing in any of the received summaries, arranged in label 
order. 

Then recovery continues by collecting the Dvs safe indications. Once the state exchange is 
safe, all labels used in the exchange are marked as safe and all associated messages are confirmed 
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Signature: 


Input: BCAST(a)p,aEA 
DVS-GPRCV(m)q,p, gE P, ME CUS 
DVS-SAFE(M)q,p, EP, MECUS 
DVS-NEWVIEW(v)p, vE V 


State: 


current € V_, init vo if p € Po, 1 else 

status € {normal, send, collect, established}, 
init normal 

content € 2°, init {} 

nextseqno € N7°, init 1 

buffer € seqof(L), init r 

safe-labels € 2°, init {} 


Output: | DVS-REGISTERp 
DVS-GPSND(m)p, mECUS 
BRCV(a)q,p, 2€ A, gE P 

Internal: ©CONFIRMp 


order € segof(L), init » 

nextconfirm € N*°, init 1 

nestreport € N°, init 1 

highprimary € G, init go if p € Po, 1 else 
gotstate, a partial function from P to S, init {} 
safe-exch C P, init {} 

registered C G, init {go} if p € Py, {} else 

delay € seqof(A), init 


Figure 7: DVS-TO-TOp, signature and states 


just as in normal processing. 


For the code, shown in Figures 7 and 8, we need the following definitions. £ = G x N>° x P 
is the set of labels, with selectors .id, l.seqno and l.origin. A is the set of messages that can be 
sent by the clients of the TO service. C = L x A is the set of possible associations between labels 
and client messages. S = 2° x seqof(L) x N>° x G is the set of summaries, with selectors x.con, 
z.ord, x.next and x.high. Given x € S, x.confirm is the prefix of x.ord such that |x.confirm| = 
min(x.next — 1,|x.ord|). If Y is a partial function from processor ids to summaries, then we 


define: 


e knowncontent(Y) = Ugedom(v) Y (q).con, 


mazprimary(Y ) = maxgedom(y)tY (q)-high}, 


maznextconfirm(Y ) = maxgedom(y) Y (q)-neat, 


e reps(Y) = {q € dom(Y) : Y(q).high = mazprimary }, 


in label order. 


chosenrep(Y ) = some element in reps(Y), 
shortorder (Y) = Y (chosenrep(Y )).ord, and 


fullorder(Y ) = shortorder(Y) followed by the remaining elements of dom(knowncontent(Y)), 


We define the system TO-IMPL to be the composition of the automata DVS-TO-TOp, for each 
p €P, and the DVS specification. with all the external actions of Dvs hidden. 
Following the approach in [17], we define the derived variables allstate, allcontent and allconfirm 


for TO-IMPL as follows. 


e We write allstate|p,g] to denote a set of summaries, defined so that x € allstate[p, g] if and 


only if at least one of the following hold: 
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Transitions: 


input BCAST(a)p 
Eff: append a to delay 


internal LABEL(a)p 

Pre: a is head of delay 
current # L 

Eff: let | be (current.id, nextseqno, p) 
content := content U {(1, a)} 
append / to buffer 
nextseqno := nextseqno + 1 
delete head of delay 


output Dvs-GPSND((I,a))p 
Pre: status = normal 
lis head of buffer 
(l,a) € content 
Eff: delete head of buffer 


input DVS-GPRCV((I, a))q,p 
Eff: content := content U {(l, a)} 
order := order-+l 


input DVS-SAFE(({I, a))¢,p 
Eff: safe-labels := safe-labels U {1} 


internal CONFIRM, 
Pre: order(nextconfirm) € safe-labels 
Eff: nextconfirm := nextconfirm +1 


output BRCV(a)¢,p 
Pre: nextreport < nextconfirm 
(order (nextreport),a) € content 
q = order (nextreport).origin 
Eff: nextreport := nextreport + 1 


Figure 8: 


x € pending|p, g]. 
(x,p) € queuelg]. 


Same ee 


Thus, allstate[p,g] consists of all the summary information that is in the state of p if p’s 
current view is g, plus all the summary information that has been sent out by p in state 
exchange messages in view g and is now remembered elsewhere among the state components 
of TO-IMPL. Notice that allstate|p,g] consists only of summaries: an ordinary message (I, a) 
is never an element of allstate[p,g]. We write allstate|g] to denote U,¢p allstate|p, g], and 


allstate to denote Uyeg allstate[g]. 


input DVS-NEWVIEW(v)p 
Eff: current := v 
neatseqno :=1 
buffer := 
gotstate := {} 
safe-exch := {} 
safe-labels := {} 


status := send 


output DVS-GPSND(z)p 
Pre: status = send 
x = (content, order, nertconfirm, highprimary) 
Eff: status := collect 


input DVS-GPRCV(2)q,p 
Eff: content := content U x.con 
gotstate := gotstate © (q, x) 


if (dom(gotstate) = current.set) A(status = collect) then 


status := established 


output DVS-REGISTERp 
Pre: status =established 
current.id ¢ registered 

Eff: registered := registered U {current.id} 
nextconfirm := maxrnextconfirm (gotstate) 
order := fullorder(gotstate ) 
highprimary := current.id 
status := normal 


input DVS-SAFE(2)q,p 
Eff: safe-erch := safe-erch U {q} 
if safe-erch = current.set then 
safe-labels := safe-labels U range(fullorder (gotstate)) 


DVS-TO-TOp, transitions 


current.id, = g and x = (contenty, order p, nextconfirm,,, highprimary »). 


For some q, current.idg = g and x = gotstate(p)g. 
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e We write allcontent for Urcatistate £-Con 


This represents all the information available anywhere that links a label with a corresponding 
data value. 


We write allconfirm for lubséatistate(x-confirm). 


For every p € P, g € G, buildorder[p, g] is defined to be a sequence of labels, initially empty; 
this variable is maintained by following every statement of processor p that assigns to order 
with another statement buildorder[p, current.idy| := order. It follows that if p registers a 
view with id g, and later leaves view g for a view with a higher view identifier, then forever 
afterwards, buildorder |p, g] remembers the value of order, at the point where p left view g. 


5.2 Correctness proof 


The correctness proof for TO-IMPL follows the approach of the proof in [17]. The pattern of 
reasoning is the same as the one used in that proof, however there are differences due to the 
distinct guarantees offered by DVS compared to those of vs. In particular Invariant 5.6, which 
corresponds to Lemma 6.18 of [17], requires a more subtle proof. 


We start by providing some auxiliary invariants. 


Invariant 5.1 (TO-IMPL) 
In any reachable state, if | € domain(allcontent) and l.origin = p then| < (current.idy, neatseqno,,p). 


Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. In the initial state, no label is associated with any message 
hence the invariant is vacuously true. 

For the inductive step, assume that the invariant is true in a reachable state s. We need 
to prove that it is true in s’ for any possible step (s,7,s’). The only step that can make the 
assertion false is the step when a new label is associated with a message from a client, hence we 
only need to consider 7 = LABEL(a)y. The code of 7 shows that the new label is less than the 
new (current.idp, nextseqno py, p) triple, since nextseqno, is incremented after being used to create 
the label. 0 


The following invariant says that when a process p has registered a view v, then any summary 
that p will create for a later view w will have its highprimary component equal to v or to a later 
view. 


Invariant 5.2 (TO-IMPL) 
In any reachable state, let x be a summary, p € P, and v,w € created such that v.id € registered, 
w.id > v.id, and x € allstate[p, w.id]. Then then x.high > v.id. 


Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. In the initial state, there are no v and w that satisfy the 
hypothesis. Hence the invariant is vacuously true. 

For the inductive step, assume that the invariant is true in a reachable state s. We need to 
prove that it is true in s’ for any possible step (s,7,s’). The steps that can make the assertion 
false are those that create new summaries or new views. 
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Let us first consider actions that create new summaries. When a new summary is created 
without modifying the high component of existing summaries in allstate|p, w.id], then the assertion 
cannot be made false. So we only need to consider action 7 =pvs-REGISTER, When current, = w.id, 
which creates the first summary x that satisfy the hypothesis in s’ (no such « existed in s). In 
this case we have that 2’ .high = w.id and by the inductive hypothesis w.id > v.id, hence we have 
that «’.high > v.id. 

Consider now actions that create new views, that is 7 =pvs-NEWVIEW(w)». We distinguish two 
cases: (1) the only registered view is vo, (2) there are registered views other than vo. In case (1) 
we have that y.high = go for any existing summary y; hence the invariant is true. In case (2) the 
invariant is true by inductive hypothesis. o 


Next we provide some other auxiliary invariants. 


Invariant 5.3 (TO-IMPL) 

In any reachable state, if x € allstate[p,g] then there exists v € created and q € v.set such 
that: (1) x.high = v.id, (2) x.high € registered ,, (3) x.ord = buildorder|q, x-high] and (4) either 
x.high = g or current.idg > v.id 


Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. In the initial state, for any such x we have that x.high = go 
and x.ord = » and thus we can take v = vp and q = p and the invariant is true. 

For the inductive step, assume that the invariant is true in a reachable state s. We need to 
prove that it is true in s’ for any possible step (s,7,s’). If x € s’.allstate[p,g], then in most 
cases, there is y € s.allstate[p, g] with y.high = x.high and y.ord = x.ord, to which we apply the 
induction hypothesis. The only case where this does not happen is when a7 =pvs-NEWVIEW(v)», 
where v.id = g, and x is the summary whose components are taken from the state of p. In this 
case, there is y € s.allstate[p, s.currentp| with y.high = x.high and y.ord = x.ord, to which we 
apply the inductive hypothesis. o 


Invariant 5.4 (TO-IMPL) 
In any reachable state, if x € allstate then there exists w € created such that x.high = w.id, and 
for all p € w.set, p € attempted|w.id]. 


Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. In the initial state, the invariant follows from the definition 
of allstate (set w = current.idp). 

For the inductive step, assume that the invariant is true in a reachable state s. We need to prove 
that it is true in s’ for any possible step (s,7,s’). The only step that we have to worry about is 
when a new summary is created. When a new summary 2 is created, x.high is set to the identifier 
of the current view, and a message has been received from everyone in the membership. O 


Invariant 5.5 (TO-IMPL) 
In any reachable state, if v € created, x € allstate and x.high > v.id then there exists p € v.set 
with current.id, > v.id. 


Proof: Fix v, x as given. Invariant 5.4 shows the existence of w € created such that x.high = w.id, 
and for all p € w.set, p € attempted[w.id]. Then Invariant 3.4 implies that there exists p € v.set 
with current-viewid|p] > v.id. But current-viewid|p] = current.idy, which yields the result. 0 
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Next we provide the crucial invariant corresponding to Lemma 6.18 of [17]. This invariant 
has a more subtle proof than the one given in Lemma 6.18 of [17]. That proof does not work 
in the setting of DVS because DVS guarantees a weaker intersection property (each primary view 
intersects only the primary views in between the preceding and the following totally registered 
primary views). The new proof also uses the fact about Dvs that once a view is totally attempted, 
no views with lower identifiers can be registered. 


Invariant 5.6 (TO-IMPL) 

In any reachable state, suppose that v € created, o € segof(L), and for every p € v.set, the 
following ts true: If current.idy > v.id then v.id € registered, anda < buildorder [p, v.id]. 

Then for every x € allstate with x.high > v.id, we have that o < x.ord. 


Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. In the initial state, the only created view is vp, and there is 
no x € allstate with x.high > go. So the statement is vacuously true. 

For the inductive step, assume that the invariant is true in a reachable state s. We need to 
prove that it is true in s’ for any possible step (s, 7,8’). So fix v € s'.created and o, and assume that 
for every p € v.set, if s’.current.idy > v.id then v.id € s'.registered, and o < s'.buildorder[p, v.id]. 

If v ¢ s.created, then 7 must be creaTevinw(v). Then v.id ¢ s' registered , for all p. Fix 
x € s'.allstate and suppose that x.high > v.id. Then Invariant 5.5 applied to s’ implies that there 
exists p € v.set with s'.current.idy > v.id; fix such a p. Then the hypothesis part of the invariant 
for s’ implies that v.id € s'.registered ,, a contradiction. It follows that v € s.created. 

As usual, the interesting steps are those that convert the hypothesis from false to true, and 
those that keep the hypothesis true while converting the conclusion from true to false. 

In this case, there are no steps that convert the hypothesis from false to true: If there is 
some p € v.set for which s.current.id, > v.id and either v.id ¢ s.registered, or o is not a 
prefix of s.buildorder |p, v.id], then also s’.current.idy > v.id (the id never decreases) and either 
v.id ¢ s'.registered , or o is not a prefix of s’.buildorder |p, v.id]. (These two cases carry over, since 
s.current.id,y > v.id implies that v.id cannot be inserted into registered, and buildorder [p, v.id] 
cannot change.) 

So it remains to consider any steps that keep the hypothesis true while converting the conclu- 
sion from true to false. Thus, we assume that if s.current.id, > v.id then v.id € s.registered,, and 
o < s.buildorder|p,v.id]. Suppose that x € s’.allstate and x.high > v.id. If also x € s.allstate 
then we can apply the inductive hypothesis, which implies that o < x.ord, as needed. So the only 
concern is with steps that produce a new summary. 

Any step that produces the new summary x by modifying an old summary 2’ € s.allstate, 
in such a way that 2’.ord < x.ord and x'.high = x.high, is easy to handle: For such a step, 
x'.high > v.id and so the inductive hypothesis implies that o < x’.ord < x.ord, as needed. So the 
only concern is with pvs-REGISTER, steps that produce a new summary x from the state-exchange 
messages of a view w sent to some processor p. Thus x.high = w.id. Let x’ be the summary of 
q' = chosenrep in s’.gotstate. We claim x’ .high > v.id. 

To prove the claim, we let v’ denote the unique element with highest view identifier among the 
elements of s’.created such that v’.id < w.id and v’ is totally registered in s’. Let v’ denote either 
uv’ or v, whichever has the higher view identifier. Invariant 3.3 shows that w.set Nv” .set # {}, no 
matter whether v” is v or v’. Fix any element q” in w.set Nv". set. 
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Recall that the precondition status = established of pvs-Newview implies that domain(s’ .gotstate,) = 
w.set, so by the code gq” € domain(s.gotstate,,). Let x be the summary s.gotstate(q'")»; we have 
x" € s.allstate|q", wid]. 

We now show that v”.id € s.registered q'- We consider two cases: 
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Then q” € v'.set so by definition of v', we have that v'.id € s.registered yy. 


2. v” =v. Because s.allstate[q”, w.id] is non-empty, we have that s.current.idg > w.id. We have 
that x.high > v.id by assumption, and x.high = w.id by the code; therefore, w.id > v.id. So 
also s.current.idg > v.id. Recall that we are in the case where the hypothesis of this invariant 
is true. Therefore, by this hypothesis (uses q” € v.set), we obtain that v.id € s.registered gu 


By Invariant 5.2 (applied with q” replacing p) we obtain x” high > vid. By the definition of 
q as a member that maximizes the high component in the summary recorded in s’.gotstate, we 
have x'.high > x" high. Therefore x’ high > v".id > v.id, completing our proof of the claim. 

If 2’ .high > v.id then we can apply the inductive hypothesis to 2’ and we are done, since 
x'.ord < x.ord. So suppose x’ .high = v.id. Note that 2’ € s.allstate|q', w.id]. By Invariant 5.3 
there must exist? q € v.set so that v.id € s. registered g, x'.ord = s.buildorder|q, v.id], and (either 
x'.high = w.id or s.current.idg > v.id). Since x’ high = v.id < x.high = w.id, the last property 
can be simplified to s.current.idg > v.id. By monotonicity of current, we have s’.current, > 
v.id. The hypothesis of this invariant says that this forces 0 < s'.buildorder|q,v.id]. Since 
x'.ord < x.ord by the code for this event, and x’.ord = s.buildorder|q, v.id] as shown above, and 
s.buildorder(q, v.id] = s'.buildorder|q, v.id] since q is not currently in view v, we get o < zx.ord, 
which is what we need. O 


Next we provide some additional auxiliary invariants. 


Invariant 5.7 (TO-IMPL) 

In any reachable state, if we have that v € created, o € segof(L), and for every p € v.set, 
v.id € registered, and o < buildorder[p, v.id], then for every x € allstate with x.high > v.id, 
o<x.ord. 


Proof: The are two possible cases: (1) x.high > v.id, (2) x.high = v.id. In case (1) we can 
apply Lemma 5.6. Consider case (2). Then we apply Lemma 5.3 to x, which gives vu’ € created 
and q’ € v’.set such that x.high = v'.id, x.high € registeredy, and x.ord = buildorder|q', x-high]. 
Since v.id = v’.id, Lemma 4.1 shows v = v’. Substituting in the facts above we see z.ord = 
buildorder[q',v.id]. Since q’ € v.set, we can apply the premise of the corollary to see that 0 < 
buildorder(q’, v.id]; that is, 0 < x.ord, as required. o 


The next invariant makes precise the fact that a label is in safe-labels,, only after it (and all 
prior labels in order,) were placed in orderg at every member q of current.sety 


Invariant 5.8 (TO-IMPL) 
In any reachable state, if 1 € safe-labels, and o is a prefix of order, that is terminated by |, then 
for all q € current.setp, o < buildorder(q, current.id>| 


Direct application of the invariant actually shows the existence of some 6 and q € é.set, but since x’ .high = 6.id 
and also x’ .high = v.id, uniqueness of view identifiers shows we may take 6 to be v itself. 
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The next lemma shows that in any summary, the ord component is closed under the relation 
“sent-before-by-one-client” . 


Invariant 5.9 (TO-IMPL) 

In any reachable state, the following is true. Assume l,l’ € Landi! € N°. Ifl,l! € domain(allcontent) 
and l.origin =l'.origin andl <l' and x € allstate and I' = x.ord(t') then there exists 1 such that 
i<@Al=z.ord(t) 


The proofs of Lemmas 5.8 and 5.9 are left as exercises. Next we show that x.confirm is a 
prefix of a known sequence. This shows the consistency of the confirmed sequence of labels at 
different places in the system. 


Invariant 5.10 (TO-IMPL) 
In any reachable state, if x € allstate then 


1. There exists v € created such that v.id < x.high and for every q € v.set, v.id € registered , 
and x.confirm < buildorder |{q, v}. 


2. x.next < length(x.ord) +1 


Proof: By induction on the length of the execution. The base case consists of proving that the 
invariant is true in the initial state. In the initial state, the only created view is vo and the only 
extant summary is ({},A,1, go); it is easy to verify that the invariant is true. 

For the inductive step, assume that the invariant is true in a reachable state s. We need to 
prove that it is true in s’ for any possible step (s, 7, 3’). 

For most of the steps, there is y in s.allstate so that y.nert = x.nert, y.ord = x.ord (and 
hence y.confirm = x.confirm) and also y.high = x.high. In these cases, the inductive hypothesis 
gives us what we want, since buildorder|q, v] increases monotonically through an execution. 

The steps that are left to consider are CONFIRMp, DVS-GPRCV((I,a))q,» ANd DVS-REGISTERp. 

Consider the case 7=conrirM,. If x is not the summary from the state of p in s’, the invariant 
follows from the inductive hypothesis. If 7 is the summary from the state of p in s’, the precon- 
dition of 7 shows that the newly confirmed message has label in s.safe-label,. By Invariant 5.8, 
taking v to be s.current, = x.high, we have part 1 of the invariant. The precondition of 7 also 
gives (x.nert — 1) € domain(x.ord), thus showing part 2 of the invariant. 

Now consider 7 =pvs-Gprcv((l,a))q,». AS before, if x is not the summary from the state of p 
in s’, the invariant follows from the inductive hypothesis. If x is the summary from the state 
of p, let y denote the summary taken from p in state s. The code shows that x.high = y.high, 
a.next = y.next, and x.ord is an extension of y.ord. By part 2 applied to y, we see that y.nezt < 
length(y.ord) +1 and therefore x.next < length(x.ord) + 1. Which proves part 2 for x; also it 
shows that x.confirm = y.confirm, so that the inductive hypothesis of part 1 applied to y proves 
part 1 for z. 

Finally consider 7 =pvs-REGISTER,. As in the other two cases, if x is not the summary from the 
state of p in s’, the invariant follows from the inductive hypothesis. If x is the summary from the 
state of p, let y denote the summary, among those in gotstate,, with the highest value for y.nezt. 
The code shows that x.nevt = w.nezt. Summary y is in s.allstate. The inductive hypothesis 
shows that y.confirm has length y.next —1, and that there is v € s.created such that v.id < y.high 
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and Vq € v.set it holds v.id € s.established, and y.confirm < buildorder[q,v]). Now let z denote 
the summary of chosenrep(gotstate), as calculated in the effect of 7. Since z.high > y.high > v.id 
(recall the definition of z as being from a representative, that is, having maximal highprimary 
among summaries in gotstate), Invariant 5.7 shows that y.confirm < z.ord. Since z.ord < x.ord 
by the code, we deduce that y.confirm is a prefix of x.ord; as length(y.confirm) = y.next —1 = 
a.nezt — 1, we have x.confirm = y.confirm. Also by the code we have x.high > y.high. Thus the 
inductive hypothesis applied to y, along with the monotonicity of the set created and the fact 
that v.id € registered, gives the invariant for 2. O 


Invariant 5.11 (TO-IMPL) 
In any reachable state, if x1, x2 € allstate and x1.high < xo.high, then x1.confirm < x9.ord. 


Proof: By Invariant 5.10, part 1, with « = x1, we have that there exists v such that v.id < x1.high 
and x1.confirm < buildorder(q,v]. By Invariant 5.7 used with o = x1.confirm since x2.high > v.id 
we have that the conclusion of Invariant 5.7 holds for 2. Hence x1.confirm < x2.ord. O 


Invariant 5.12 (TO-IMPL) 
In any reachable state, for any x,2' € allstate, either x.confirm < 2'.confirm or x'.confirm < 
x.confirm. 


Proof: Without loss of generality, we can assume that x.high < 2’ high. From Invariant 5.11, we 


have that both x.confirm and z’.confirm are prefixes of x’.order. O 


To prove that TO-IMPL implements TO, we define a function from the reachable states of TO- 
IMPL to the states of TO and prove that it is an abstraction function. This function, called Fro, 
is defined exactly as in [17] and it is given in Figure 9. 


Let s be a state of TO-IMPL. The state t = Fro(s) of TO is the following. 


1. t.queue = applyall((s.allcontent, origin), s.allconfirm), 
where the selector origin is regarded as a function from labels to processors. 


2. t.next|p] = s.neat-report.,. 


3. t.pending[p| = applyall(s.allcontent, b) - s.delay,, where 6 is the sequence of labels such that 


(a) range(b) is the set of labels J such that l.origin = p, (l,a) € s.allcontent for some a, 
and 
| ¢ range(s.allconfirm). 

(b) 6 is ordered according to the label order. 


Figure 9: The abstraction function Fro. 


In order to prove that Fro is an abstraction function we need to prove that (a) for any initial 
state s of TO-IMPL we have that Fro(s) is an initial state of TO, and that (b) for any possible 
step a of TO-IMPL there exists an execution fragment a of TO such that the trace of a is equal to 
the trace of 7, that is, w and 7 have identical externally observable behaviors. Lemmas 5.13 and 
5.14 prove this. 
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Lemma 5.13 If s is an initial state of TO-IMPL then F(s) is an initial state of TO. 


Proof: Let s be the initial state of TO-IMPL. Let t = Fro(s). By the definition of Fro we have 
that t.queue = X, t.next[p] = s.neat-report, = 1 for any p and that t.pending = A. Hence ¢ is an 
initial state of TO. 0 


Lemma 5.14 Lets be a reachable state of TO-IMPL, Fro(s) a reachable state of TO, and (s, 7, s’) 
a step of TO-IMPL. Then there is an execution fragment a of TO that goes from Fro(s) to Fro(s’), 
such that trace(a) = trace(m). 


Proof: By case analysis based on the type of the action 7. Define t = Fro(s) and t! = Fro(s’). 


1. m = BCAST(a)p 


Since m is an input to TO, 7 is enabled in ¢t. The effect of a shows that s’.allconfirm = 
s.allconfirm, s’.allcontent = s.allcontent, and s’.pending|p] = s.pending[p| + a. This implies 
that t'.pending|[p| = t.pending[p] + a, thus showing that (t, 7, t’) is a step of TO. Hence we set 
a=. Clearly trace(a) = trace(z). 


2. 7 = LABEL(a)p 


We to show that t = t’. The effect of a shows that t’.allconfirm = zx.allconfirm, and 
t'.allcontent is the union of t.allcontent with (I',a) where I’ = (t.currentp, x.neatseqno yp); 
by Invariant 5.1, this new label /’ is greater than all labels in the domain of x.allcontent. Thus 
let us consider the sequence of labels o’ (arranged in label order) such that range(o’) is the 
set of labels / such that l.origin = p, (l,a’) € x’.allcontent for some a’, and 

| ¢ range(x'.allconfirm). We see that o’ is related to the sequence o (defined the same 
way but using s instead of s’) by o’ = 0 +I’. Therefore applytoall(s'.allcontent,o') = 
applytoall(s.allcontent,a) +a. On the other hand, the precondition of 7 shows that a is 
the head of s.delay,,, and so the effect of means s.delay, = a+ <x'.delay,. Thus, t’.pending|[p| 
is the same as t.pending[p], because in the concatenation that defines this component, the 
element a is simply transferred from suffix to prefix. Therefore t’ = t. Hence we set a = t. 


3. 7 = CONFIRMp 
Clearly the effect of 7 shows s.allcontent = s'.allcontent. 


If s.nextconfirm, < length(s.allconfirm) then Invariant 5.12 and the effect of 7 shows that 
s'.allconfirm = s.allconfirm, so that t = t’. In this case we set a = t. 


Otherwise s.neztconfirm, = length(s.allconfirm)+1, so the effect of 7 shows that s'.allconfirm = 
x.allconfirm - (l) where | = s.ordery(s.nextorder,). Let q = l.origin and a = s.allcontent(l). 
We claim that (t,ro-orper(a,q),t’) is a step of TO. 


We first show that ro-orpeR(a,q) is enabled in t. We have | € domain(s.allcontent) and 
l ¢ setof (s.allconfirm); this means that a is an element of the sequence t.pending[q|. Also by 
Invariant 5.9, any lower label with origin q is in s.confirm, and so in s.allconfirm. Since the 
sequence o used to define t.pending|q] is arranged by label, we see that | is the head of o, 
and so a is the head of t.pending|q], as required. Further, the equation above for t’.allconfirm 
shows that t'.queue = t.queue + (a,p), and this is what is needed to show that 7 takes ¢ to t’. 


Hence we set a@ =To-orDER(a,q). Clearly trace(a) = trace(m) = X. 
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4. 17 = DVS-GPRCV(s)q,p 


In some cases this may change the value of neztconfirm,, but in every situation it leaves 
allconfirm unchanged (it only moves nextconfirm q to a value already somewhere in allstate). 
Thus ¢’ = t. Hence we set a = t. 


5D. T = BRCV(a) p,q 


We need to show that 7 is enabled in ¢ as an action of TO. This is immediate from the fact that 
m is enabled in s as an action of TO-IMPL. Similarly, the effect corresponds (only nextreport ¢ 
is altered). 


Hence we set a@ =Brcv(a)q,». Clearly trace(a) = trace(m) = X. 


6. Remaining actions. 


The other actions leave t' = t. Hence we set a = t. 


0 


Lemmas 5.13 and 5.14 prove that Fro is an abstraction function from TO-IMPL to TO and 
thus the following theorem holds (this is a standard inference, cf. [29]). 


Theorem 5.15 Every trace of TO-IMPL is a trace of TO. 


6 Discussion 


We presented a specification for a dynamic primary view group communication service and an 
algorithm that formally implements the service, and we showed the utility of our new specification 
by using it to implement a totally ordered broadcast. This work deals entirely with safety proper- 
ties; future work could consider performance and fault-tolerance properties using the conditional 
performance analysis as presented in [17]. It also remains to study other applications of our Dvs 
specification, such as replicated data applications and load-balancing applications. 

Another interesting exploration direction considers variations on the DVS specification, for 
example, one in which the state exchange at the beginning of a new view is supported by the 
dynamic view service. We are currently studying variations on our specifications that are more 
specifically tuned to systems like Isis and Ensemble. In particular, we would like to understand 
the power of the Isis requirement that processes that move together from one view to the next 
receive exactly the same messages in the first view, especially for coherent-data applications. 

In a related work [11] we also have investigated a generalization of the DVS service to dynamic 
sets of primaries rather than individual primaries, in order to tolerate transient failures during a 
particular view. 
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